Video transmission control method applied to augmented reality and related equipment

Abstract

The embodiment of the invention discloses a video transmission control method applied to augmented reality and related equipment. The method comprises the following steps: receiving a video acquisition request which is sent by a client and contains the movement information of an experiencer; candidate views matched with the moving information of the experiencer are predicted through a plurality of preset view prediction models, and moving rear view port positions determined by different view prediction models based on the moving information of the experiencer are different when the candidate views are predicted; the predicted image contents of the candidate views are correspondingly different; obtaining an experience quality value predicted for each candidate view, and performing fusion processing on the candidate views based on the experience quality value corresponding to each candidate view to obtain a to-be-transmitted view; and transmitting the view to be transmitted to the client. Even if the movement track of the experiencer suddenly changes or other factors occur, the user experience in the augmented reality scene can still be improved.

Description

Technical Field

[0001] The present application relates to the technical field of extended reality, and in particular to a video transmission control method and related equipment applied to extended reality. Background Art

[0002] Extended Reality (XR) is a general term for Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). It uses wearable devices and computer technology to bring users an environment that combines the real and virtual, and human-computer interaction, to achieve an "immersive" experience of seamless transition between the virtual world and the real world. In the actual application of extended reality, user experience is an aspect that deserves great attention. At present, there are still problems such as inaccurate video picture prediction due to sudden changes in the user's movement trajectory or other factors, which in turn leads to poor user experience. Summary of the invention

[0003] The embodiments of the present application respectively provide a video transmission control method applied to augmented reality, a video transmission control device applied to augmented reality, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, an embodiment of the present application provides a video transmission control method for extended reality, the method comprising: receiving a video acquisition request containing experiencer movement information sent by a client node; predicting candidate views matching the experiencer movement information respectively through a plurality of preset view prediction models, wherein different view prediction models have different viewport positions after movement determined based on the experiencer movement information when predicting candidate views, so that the predicted picture contents of each candidate view are correspondingly different; obtaining an experience quality value predicted for each candidate view, and performing fusion processing on the candidate views based on the experience quality values ​​corresponding to each candidate view to obtain a view to be transmitted; and transmitting the view to be transmitted to the client node.

[0005] In a second aspect, an embodiment of the present application provides a video transmission control device for extended reality, the device comprising: a receiving module, configured to receive a video acquisition request containing experiencer movement information sent by a client node; a prediction module, configured to predict candidate views matching the experiencer movement information respectively through a plurality of preset view prediction models, wherein different view prediction models have different viewport positions after movement determined based on the experiencer movement information when predicting candidate views, so that the predicted picture contents of each candidate view are correspondingly different; an acquisition module, configured to obtain an experience quality value predicted for each candidate view, and perform a fusion process on the candidate views based on the experience quality values ​​corresponding to each candidate view to obtain a view to be transmitted; and a transmission module, configured to transmit the view to be transmitted to the client node.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the steps in the video transmission control method for extended reality as described above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a computer, the computer executes the steps in the video transmission control method applied to extended reality as described above.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the video transmission control method applied to extended reality as described above.

[0009] In the technical solution provided in the embodiments of the present application, after receiving the video acquisition request sent by the client, the server node of the extended reality predicts the candidate views respectively through the preset multiple view prediction models. Since different view prediction models consider different viewport positions after movement for candidate view prediction, the picture contents of the candidate views predicted by different view prediction models are also different accordingly. Based on this, the experience quality value corresponding to each candidate view is subsequently predicted, and each candidate view is fused based on the experience quality value of each candidate view to obtain the view to be transmitted. This is equivalent to the viewport position corresponding to the view to be transmitted being a video picture matched with a more accurate viewport position obtained by fusion of the predicted multiple viewport positions. Even if there is a sudden change in the experiencer's movement trajectory or other factors, the video picture corresponding to the accurate viewport can be predicted, thereby transmitting the video to be transmitted to the client node for extended reality related display, which can enable the experiencer to obtain a higher user experience.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of an implementation environment of an example of the present application;

[0012] Figure 2 is a flow chart of a video transmission control method applied to augmented reality shown in an exemplary embodiment of the present application;

[0013] Figure 3 exist Figure 2 A flowchart of a video transmission control method applied to augmented reality further proposed on the basis of the illustrated embodiment;

[0014] Figure 4 is Figure 2 A flowchart of another video transmission control method applied to augmented reality further proposed on the basis of the illustrated embodiment;

[0015] Figure 5 is Figure 2 A flowchart of another video transmission control method applied to augmented reality further proposed on the basis of the illustrated embodiment;

[0016] Figure 6 An example of an architecture diagram of a stereoscopic video transmission system is provided;

[0017] Figure 7 It is for Figure 6 The detailed flow chart of the stereoscopic video transmission system shown in the figure is to implement stereoscopic video transmission;

[0018] Figure 8 is a block diagram of an exemplary video transmission control device applied to augmented reality of the present application;

[0019] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0023] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0024] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0025] The first thing we need to understand is that extended reality (hereinafter referred to as XR) is a general term for virtual reality (hereinafter referred to as VR), augmented reality (hereinafter referred to as AR) and mixed reality (hereinafter referred to as MR). It uses wearable devices and computer technology to bring the experiencer an environment that combines reality and virtuality and human-computer interaction.

[0026] VR, also known as virtual reality or spiritual technology, includes computers, electronic information, and simulation technologies. Its basic implementation method is based on computer technology, utilizing and integrating the latest scientific and technological developments such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. With the help of computers and other equipment, a realistic three-dimensional visual, tactile, olfactory and other sensory experiences are created in the virtual world, giving people a sense of being immersed in the virtual world.

[0027] AR is a technology that cleverly integrates virtual information with the real world. It widely uses multimedia, three-dimensional modeling, intelligent interaction, sensing and other technical means. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, video, etc., and applies them to the real world. After the real environment and virtual objects overlap, they can exist simultaneously in the same screen and space. It can be seen that the two types of information complement each other, thereby achieving "enhancement" of the real world.

[0028] MR is a further development of VR technology. By introducing real-world scene information into a virtual environment, MR builds an interactive feedback information loop between the virtual world, the real world and the user to enhance the realism of the user experience.

[0029] In general, XR is an "immersive" experience that seamlessly switches between the virtual world and the real world by integrating at least one of VR, AR, and MR visual interaction technologies. However, it should be noted that with the further development of science and technology, other visual interaction technologies besides VR, AR, and MR may appear in the future, so the XR mentioned in this application should also include these other visual interaction technologies that will appear in the future.

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0032] The technical solutions provided in the embodiments of the present application involve both the XR field and the machine learning field. The technical solutions provided in the present application will be introduced in detail below.

[0033] First see Figure 1 , Figure 1 It is a schematic diagram of an implementation environment of an example of the present application, and the application environment is specifically an XR application scenario, which includes a client node 110 and a server node 120. The client node 110 includes a wearable device, which is worn on the experiencer himself to sense the movement of the experiencer and display the virtual environment screen to the experiencer. An XR client can be installed in the wearable device, so that the perceived movement information of the experiencer can be transmitted to the server node 120 through the XR client, and a request can be made to obtain the video content. The client node 110 also includes a terminal device, on which an XR client runs, and the wearable device can also transmit the experiencer's movement information perceived by itself to the terminal device, so that the terminal device reports the experiencer's movement information to the server node 120 and requests to obtain the video content at the same time.

[0034] The server node 120, as the provider of the virtual environment, stores the background data of the client node 110 and all the video content of the virtual environment. It predicts the position and / or posture of the experiencer after movement based on the movement information reported by the client node 110, and then sends the video content corresponding to the position and / or posture after movement to the client node 110. The terminal device in the client node 110 renders the video content and sends it to the wearable device for display, or forwards the video content to the wearable device for rendering and display, so as to display the virtual environment for the experiencer to see, thereby providing the experiencer with a use environment that combines reality and virtuality.

[0035] It is understandable that the wearable device can be a smart device such as smart glasses, smart watches, etc., and the terminal device can be, for example, Figure 1 The devices shown include, but are not limited to, smart phones, personal computers (PCs), all-in-one machines, etc. The server node 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, which are not limited here. The terminal device and the server can be directly or indirectly connected via wired or wireless communication to enable data interaction between the client node 110 and the server node 120, which is not limited here.

[0036] It is also necessary to understand that cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. The background services of the cloud technology network system require a large amount of computing and storage resources, such as video websites, image websites and more portal websites, which can only be achieved through cloud computing. Cloud computing is the distribution of computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, and expanded at any time. Therefore, when Figure 1 When the server node 120 shown is deployed on a cloud server, it can provide stable and reliable data services for the implementation of XR application scenarios, thereby providing experiencers in the XR application scenarios with an "immersive" experience of seamless transition between the virtual world and the real world.

[0037] In the actual application of XR, user experience is an aspect that deserves great attention. At present, there are still problems of inaccurate video picture prediction due to sudden changes in the experiencer's movement trajectory or other factors, which leads to poor user experience. For example, if the experiencer's movement trajectory changes suddenly, after the client node 110 reports the experiencer's movement information to the server node 120, the experiencer's position and / or posture predicted by the server node 120 after the movement will be different from the experiencer's actual position after the movement, resulting in the corresponding video picture prediction in the virtual environment obtained by the server node 120 being inaccurate. Sending inaccurate video pictures to the client node 110 for display may cause the experiencer to feel dizzy, and more seriously, it will affect the interactivity of the experiencer.

[0038] To solve the above technical problems, the embodiments of the present application respectively provide a video transmission control method applied to extended reality, a video transmission control device applied to extended reality, an electronic device, a computer storage medium, and a computer program product. These embodiments will be described in detail below.

[0039] See also Figure 2 , Figure 2 FIG. 1 is a flowchart of a video transmission control method for augmented reality shown in an exemplary embodiment of the present application. Figure 1 The application scenario shown is specifically implemented by the server node 120 in the application scenario. Of course, the method can also be applied to other application scenarios and specifically implemented by the server in other application scenarios, and the embodiments of the present application are not limited to this.

[0040] The following describes the content of this method in detail by taking the server node as the specific execution subject of this method. Figure 2 As shown, in an exemplary embodiment, the video transmission control method applied to augmented reality includes S210-S240, which are described in detail as follows:

[0041] S210: Receive a video acquisition request including movement information of an experiencer sent by a client node.

[0042] The video acquisition request containing the experiencer's movement information sent by the client node to the server node is used to request the server node to obtain a virtual environment picture that matches the experiencer's movement state.

[0043] It should be noted that in different actual XR application scenarios, the specific content of the experiencer's movement information may also be different, and the information contained in the experiencer's post-movement state is also different accordingly. For example, in a scenario where the virtual environment video is a two-dimensional video, such as a panoramic video scene, the experiencer's movement information includes the experiencer's position information, and the experiencer's post-movement state also includes the experiencer's position information after movement. In a scenario where the virtual environment video is a three-dimensional video, such as a stereoscopic video scene, the experiencer's movement information includes the experiencer's posture information in addition to the experiencer's position information, and the experiencer's post-movement state also includes the experiencer's position information and posture information after movement.

[0044] It should also be noted that, since the experiencer's state after movement is future information, for the client node, the experiencer's movement information carried in the video acquisition request is the experiencer's current location related information. For the server node, after receiving the video acquisition request, it needs to predict the experiencer's position and / or posture after movement based on the experiencer's current location related information, and then return the virtual video content matching the predicted position and / or posture to the client node for display and other processing.

[0045] S220, predicting candidate views that match the experiencer's movement information respectively through a plurality of preset view prediction models, wherein different view prediction models have different viewport positions after movement determined based on the experiencer's movement information when predicting the candidate views, so that the picture contents of the predicted candidate views are correspondingly different.

[0046] As mentioned earlier, user experience is an aspect that deserves great attention in the actual application of XR. At present, there is still the problem of inaccurate video picture prediction due to sudden changes in the experiencer's movement trajectory or other factors, which in turn causes poor user experience. The fundamental reason for the inaccurate video picture prediction is that it is difficult for the server node to accurately predict the position and / or posture of the experiencer after movement.

[0047] To solve this problem, the embodiment of the present application presets and deploys multiple view prediction models on the server node to respectively predict candidate views that match the experiencer's movement information carried in the video acquisition request. It can be understood that the candidate view that matches the experiencer's movement information is the video content corresponding to the virtual environment that matches the experiencer's movement information, obtained by the view prediction model after predicting the experiencer's position and / or posture after movement based on the experiencer's movement information.

[0048] In order to save transmission bandwidth resources, in this embodiment, the virtual video content transmitted by the server node to the client node is not a full-view video, but a video at the viewport position of the experiencer. The viewport of the experiencer can be understood as the picture area that the experiencer can currently see, or as the range displayed by the virtual video screen projection in the program window, and the candidate view is the view screen content that matches the experiencer's viewport. Since different view prediction models predict different positions and / or postures of the experiencer after movement based on the experiencer's movement information, it can be understood that different video prediction models also determine different viewport positions after movement based on the experiencer's movement information when predicting candidate views. Therefore, the picture content of the candidate views predicted by each video prediction model is also correspondingly different.

[0049] It should be understood that each view prediction model should be used to predict the candidate view that best matches the experiencer's movement information. However, since the model parameters of different view prediction models are different, the viewport positions after movement determined by each view prediction model are different, so that the picture contents of the candidate views predicted by each view prediction model are different.

[0050] Exemplarily, different view prediction models can be different types of machine learning models. For example, the view prediction model can be a time series prediction model, such as LSTM (Long Short-Term Memory), RNN (Recurrent Neural Network) and other models, which predict the future viewport position through historical viewport position changes, and then obtain the candidate view corresponding to the predicted future viewport position. The view prediction model can also be a target tracking model based on video content, such as Tensor Flow (a deep learning model that uses a graph to represent computing tasks, all calculations in the model are converted into nodes on the computing graph, and the edges between the nodes describe the computing dependencies between the nodes), DeepSORT (a multi-target tracking algorithm model based on deep learning), etc., which predicts the future viewport position through changes in the video screen that the experiencer has seen in the past, and then obtains the corresponding candidate view.

[0051] As another exemplary implementation, different view prediction models may also be multiple models obtained by performing different model structure improvements on the basis of a certain machine learning model. This embodiment does not limit the specific model structures of different view prediction models.

[0052] S230: Obtain a predicted experience quality value for each candidate view, and perform a fusion process on the candidate views based on the experience quality values ​​corresponding to the candidate views to obtain a view to be transmitted.

[0053] As mentioned above, user experience is an aspect that deserves great attention in XR applications. This enables this embodiment to predict the corresponding quality of experience (QoE) value for each candidate view after predicting multiple candidate views based on multiple view prediction models. In this embodiment, the quality of experience value refers to the subjective feeling of the experiencer on the XR application scenario, which can also be understood as the subjective feeling of the XR experience felt by the experiencer.

[0054] Considering that the candidate view with the highest predicted quality of experience value may still have a certain deviation from the actual user experience in actual applications, this embodiment does not directly select the candidate view with the highest quality of experience value as the view to be transmitted, but after obtaining the QoE value predicted for each candidate view, the candidate view is fused based on the QoE value predicted for each candidate view. It can be understood that relevant content information is obtained from multiple candidate views to form the optimal video content, so as to obtain the view to be transmitted. The optimal video content is understood to enable the experiencer to obtain the best user experience after being transmitted to the client. For example, the view to be transmitted can be obtained by sampling the corresponding video content combination from each candidate view, or each candidate view can be downsampled to different degrees to obtain a corresponding downsampled image, and then each downsampled image is fused to obtain the view to be transmitted. Other fusion methods can also be used, which will not be described in detail here.

[0055] S240: Transmit the view to be transmitted to the client node.

[0056] The server node transmits the view to be transmitted to the client node, so that the client node performs processing related to the view to be transmitted, such as rendering, displaying, forwarding, etc., so that the experiencer can get a good immersive XR experience.

[0057] As can be seen from the above, in the technical solution provided by this embodiment, after receiving the video acquisition request sent by the client node, the server node of the extended reality predicts the candidate views respectively through the preset multiple view prediction models. Since the viewport positions after movement considered by different view prediction models for candidate view prediction are different, the picture contents of the candidate views predicted by different view prediction models are also different accordingly. Based on this, the experience quality values ​​corresponding to each candidate view are subsequently predicted, and each candidate view is fused based on the experience quality values ​​of each candidate view to obtain the view to be transmitted. This is equivalent to the viewport position corresponding to the view to be transmitted being the video content matched by the more accurate viewport position obtained by fusion of the predicted multiple viewport positions. Even if there is a sudden change in the experiencer's movement trajectory or other factors, the video content corresponding to the accurate viewport can be predicted, thereby transmitting the video to be transmitted to the client node for extended reality related display, which can enable the experiencer to obtain a higher user experience.

[0058] Figure 3 exist Figure 2 A flowchart of a video transmission control method for augmented reality is further proposed based on the embodiment shown. Figure 3 As shown, the process of performing fusion processing of candidate views based on the experience quality values ​​corresponding to each candidate view in S230 includes S231-S233, which are described in detail as follows:

[0059] S231 , sorting the candidate views in order of experience quality values ​​from high to low or from low to high.

[0060] This embodiment is based on the idea of ​​performing downsampling processing on the corresponding candidate views to different degrees according to different QoE values, and then fusing the obtained downsampled images, to achieve the fusion of candidate views to obtain the view to be transmitted. Therefore, it is necessary to sort the multiple candidate views obtained in S220 in the order of high to low or low to high QoE values ​​to obtain the corresponding candidate view sequence.

[0061] S232 , down-sampling the corresponding candidate views according to the sorting order of the candidate views to obtain down-sampled images corresponding to the candidate views.

[0062] In this embodiment, the corresponding downsampling frequency is first determined according to the sorting order of each candidate view, and then the corresponding candidate view is downsampled based on the determined downsampling frequency, so as to obtain the downsampled image corresponding to each candidate view.

[0063] It should be noted that downsampling is also called downsampling, and its sampling frequency is lower than 2 times of the signal sampling frequency of the candidate view, which can be generally understood as the process of reducing the image. The lower the downsampling frequency, the lower the amplitude of the image reduction; the higher the downsampling frequency, the higher the amplitude of the image reduction. Therefore, in this embodiment, the sorting order represents that the candidate view with a higher experience quality value has a lower corresponding downsampling frequency, which means that the degree of reduction for the candidate view with a higher experience quality value is also lower. For example, if the candidate views are sorted in order from high to low experience quality values, the candidate view with a higher ranking has a lower downsampling frequency and a corresponding lower downsampling amplitude; if the candidate views are sorted in order from low to high experience quality values, the candidate view with a higher ranking has a higher downsampling frequency and a corresponding higher downsampling amplitude.

[0064] S233: Fusing downsampled images corresponding to the candidate views to obtain a view to be transmitted.

[0065] In this embodiment, based on different methods of predicting candidate views that match the experiencer's movement information in S220, different methods of fusing downsampled images are used, thereby obtaining different views to be transmitted.

[0066] For example, as an exemplary implementation, after determining the viewport information after the movement according to the movement information of the experiencer, S220 generates video content matching the determined viewport information according to the pre-stored full-view video to obtain the corresponding candidate view. For example, if the pre-stored full-view video is a two-dimensional video content, the video screen matching the determined viewport information can be directly intercepted from the pre-stored full-view video as the corresponding candidate view; if the pre-stored full-view video is a stereoscopic video content, a three-dimensional virtual scene matching the determined viewport information can be first obtained based on the pre-stored full-view video, and then the three-dimensional virtual scene can be rendered into a two-dimensional image as the corresponding candidate view.

[0067] In this exemplary embodiment, since each candidate view is directly obtained from a pre-stored full-view video and matches the viewport information predicted by the video prediction model, after downsampling each candidate view to obtain a corresponding downsampled image, the downsampled images corresponding to each candidate view are directly fused to obtain a corresponding fused image, and this fused image is used as the view to be transmitted. Among them, fusing each downsampled image can be understood as superimposing each downsampled image to obtain a fused image, that is, in this case, the fused image is an image. Alternatively, multiple downsampled images can be taken as a whole, and this whole is called a fused image, that is, in this case, the fused image contains multiple images, and this embodiment does not limit the specific fusion method of the downsampled images.

[0068] In another exemplary embodiment, after determining the viewport information after the movement according to the experiencer's movement information, S220 first obtains the video block that matches the determined viewport information, and then uses the video block as the candidate view corresponding to the viewport information. It should be understood that the server node divides the full-view video screen into multiple video blocks in advance, and each video block corresponds to a code stream that can be independently decoded during transmission, and prepares a low-quality full-view video code stream at the same time. The low-quality full-view video can be understood as a full-view video that has been processed with reduced quality. For example, by pre-encoding the full-view video with a lower encoding bit rate or reducing the resolution of the full-view video, the full-view video that has been processed with reduced quality can be obtained. And it can be seen that the picture quality of each video block is higher than the picture quality of the full-view video that has been processed with reduced quality.

[0069] In this exemplary embodiment, the downsampled images corresponding to each candidate view can be fused in the same manner as the above embodiment to obtain a fused image, and then a full-view video with reduced quality is obtained, and the fused image and the full-view video with reduced quality are used together as the view to be transmitted.

[0070] It can be seen that in the technical solution provided in this embodiment, based on the idea of ​​downsampling the corresponding candidate views to different degrees according to different QoE values ​​and then fusing the obtained downsampled images, the fusion of candidate views is realized to obtain the view to be transmitted, which can feed back a video picture with theoretically optimal user experience to the client node, so that the experiencer can obtain the theoretically optimal XR application experience.

[0071] Figure 4 Also in Figure 2 A flowchart of a video transmission control method for augmented reality is further proposed based on the embodiment shown. Figure 4 As shown, the process of obtaining the predicted experience quality value for each candidate view in S220 includes S221-S222, and the method further includes S410-S420 after S240, which are described in detail as follows:

[0072] S221, inputting the historical experience quality of each candidate view into an experience quality prediction model, wherein the historical experience quality value is determined based on at least one indicator of MTP delay, viewport offset distance, and video subjective perception quality.

[0073] This embodiment uses an experience quality prediction model to predict the experience quality value of each candidate view. It can be understood that the experience quality prediction model is essentially a machine learning model. For example, the experience quality value prediction model can be a time series prediction model. By inputting historical values ​​of indicators related to the experience quality value of the candidate view into the experience quality prediction model, the experience quality prediction model can predict the future values ​​of these indicators, and then calculate the experience quality value according to the future values ​​of these indicators, so as to obtain the experience quality value of the candidate view predicted by the experience quality prediction model.

[0074] The historical experience quality value of the candidate view is determined based on at least one of the following indicators: MTP (Motion To Photons) delay, viewport offset distance, and subjective video perception quality. These indicators are indicators related to the experience quality value. Among them, the MTP delay represents the length of time it takes from the experiencer to move to the corresponding view screen display. If the MTP delay table is too large, it is easy to cause dizziness to the experiencer, so the MTP delay is used as an indicator related to the experience quality value. The viewport offset distance refers to the offset distance between the actual viewport and the predicted viewport. The predicted viewport is also the viewport information predicted by the experience quality prediction model based on the experiencer's movement information. If the viewport offset distance is too large, the picture will be unbalanced, so the viewport offset distance is also used as an indicator related to the experience quality value. Video quality can also be called subjective video perception quality, which is used to measure the subjectively perceived video quality of the experiencer. This is an important indicator related to the experience quality value.

[0075] For example, the subjective perceived quality of a video can be measured by at least one of a PSNR (Peak signal-to-noise ratio) indicator and a SSIM (Structural Similarity) indicator. PSNR refers to the logarithm of the mean square error between the coded image and the corresponding view slice relative to a specified value, where the specified value refers to the square of the maximum value of the signal and can be expressed as “(2 n -1) 2 ", where n represents the number of bits of the sampled value, so it can also be used to evaluate the similarity between images. The SSIM indicator refers to the picture similarity between the coded image and the corresponding view slice.

[0076] S222: Obtain the experience quality value predicted by the experience quality prediction model based on the historical experience quality value of each candidate view.

[0077] The experience quality prediction model predicts the future values ​​of these indicators based on the indicators related to the historical experience quality values ​​of each candidate view, and then calculates the corresponding experience quality value according to the future values ​​of these indicators, so as to use this experience quality value as the experience quality value predicted by the experience quality prediction model. Exemplarily, when the subjective perceived quality of the video is specifically measured by the PSNR indicator and the SSIM indicator, the subjective perceived quality of the video can be the sum of the two indicators, or the weighted sum of the two indicators, or the value obtained by calculating the two indicators using other calculation methods, and this embodiment does not limit this.

[0078] S410: After transmitting the view to be transmitted to the client node, obtain a real experience quality value fed back by the client node.

[0079] After transmitting the video to be transmitted to the client node, this embodiment also obtains the real experience quality value fed back by the client node. It can be understood that the real experience quality value is still calculated based on the aforementioned indicators. For example, if the indicators used to measure the experience quality value include MTP delay, viewport offset distance and video subjective perception quality indicators, the client node also collects these indicators accordingly during the process of display-related processing of the video to be transmitted, and finally calculates the corresponding real experience quality value, and feeds back the real experience quality value to the server node. Alternatively, the client node can also feed back the indicators collected by itself to the server node, so that the server node obtains the real experience quality value by performing relevant calculations on the received feedback indicators.

[0080] S420: Update the experience quality prediction model based on the actual experience quality value.

[0081] It should be understood that, generally speaking, machine learning models are trained offline to achieve a certain model effect before being used online. This embodiment is equivalent to building an experience pool to maintain the real QoE value of the view that has been transmitted to the client node, and updating the experience quality prediction model online based on the maintained real QoE value, thereby continuously improving the prediction accuracy of the experience quality prediction model.

[0082] It can be seen that the technical solution provided by this embodiment obtains the experience quality value of the candidate view through specific indicators, so that the experience quality value of the candidate view can accurately reflect the level of user experience that the experiencer can obtain to a high degree. This embodiment also updates the experience quality prediction model online by maintaining the real experience quality value of the transmitted view, thereby continuously improving the prediction accuracy of the experience quality prediction model during the XR application process, which also provides important technical support for further improving the XR application experience of the experiencer.

[0083] Figure 5 Still in Figure 2 A flowchart of a video transmission control method for augmented reality is further proposed based on the embodiment shown. Figure 5 As shown, the process of transmitting the view to be transmitted to the client node in S240 includes S510-S540, which are described in detail as follows:

[0084] S510, slicing the view to be transmitted to obtain a corresponding plurality of view slices.

[0085] The various embodiments described above are all optimized for the transmission content of the XR application scenario. Considering that the amount of data that needs to be transmitted in the XR application scenario is relatively large, when faced with changes in the transmission link of the user device and fluctuations in the wireless signal, it is easy to cause the transmission throughput to be greater than the bottleneck bandwidth, resulting in network congestion, thereby causing delays in the client node and bringing bad effects to the interactivity of the experiencer. To solve this problem, this embodiment optimizes the XR application scenario from the perspective of transmission bit rate.

[0086] In this embodiment, the video to be transmitted is first sliced ​​to divide the view to be transmitted into a plurality of view slices. In this embodiment, the scale of the slicing process of the video to be transmitted can be preset, for example, the slicing process of the video to be transmitted is preset to be small in scale, which means that the number of view slices obtained by slicing the video to be transmitted is small.

[0087] S520: Perform pre-coding processing on each view slice, and determine a video quality indicator corresponding to the view to be transmitted based on the picture difference between the obtained coded image and the corresponding view slice.

[0088] After obtaining the corresponding multiple view slices based on the view to be transmitted, this embodiment performs pre-coding processing on each view slice according to the preset coding rate to obtain a coded image. It can be understood that the pre-coding process is still the process of encoding the view slices according to the preset coding rate, but the coded image obtained by pre-coding is not necessarily transmitted to the client. Instead, it is necessary to determine whether to send the coded image directly to the client based on the corresponding video quality index, or to encode the video to be transmitted based on the new coding rate and then send it to the client node. It should also be understood that the preset coding rate can be a dynamic coding rate preset in the congestion control algorithm configured in the video transmission protocol deployed in the server node. Of course, in some exemplary embodiments, the preset coding rate can also be a preset fixed rate value, which is not limited in this embodiment.

[0089] The video quality index corresponding to the view to be transmitted can still use at least one of the PSNR index or the SSIM index, which are all obtained based on the picture difference between the encoded image and the corresponding view slice. It can be understood that the SSIM index refers to the picture similarity between the encoded image and the corresponding view slice, and the PSNR refers to the logarithm of the mean square error between the encoded image and the corresponding view slice relative to a specified value, which refers to the square of the maximum signal value. When the video quality index uses the PSNR index and the SSIM index, the video quality index can be the sum of the two indicators, or the weighted sum of the two indicators, or the value obtained by calculating the two indicators using other calculation methods, and this embodiment does not limit this.

[0090] Alternatively, in some exemplary embodiments, the video to be transmitted may be directly encoded at a preset encoding bit rate to obtain an encoded image, and then the corresponding video quality index may be determined based on the picture difference between the encoded image and the original video content.

[0091] S530: If the video quality index does not meet the preset index condition, determine a bit rate that matches the video quality index.

[0092] If the video quality index does not meet the preset index condition, it means that the bit rate of pre-encoding the multiple view slices is too low, so it is necessary to re-determine the bit rate that matches the video quality index, which should be higher than the bit rate corresponding to the pre-encoding.

[0093] For example, WebRTC (Web Real-Time Communication) technology is the main technology for transmitting XR video. WebRTC is a technology that drives real-time communication (including voice, video and arbitrary data) methods directly in a Web browser. It is an open source technology that can perform real-time communication work in a web page without installing any plug-ins in the browser. Instead, it directly adds real-time media communication between the browser and the device. WebRTC itself is designed for audio and video calls, supports two-way video communication, adjusts the bit rate through bandwidth prediction, and is suitable for real-time scenarios. Therefore, WebRTC's congestion control method can be used to achieve transmission bit rate control in XR application scenarios, such as GCC (Google Congestion Control), BBR (Bottleneck Bandwidth and Round-trip propagation time), PCC (Performance-oriented Congestion Control), etc., which are not limited here. Among them, the core idea of ​​GCC is to control the sending rate by predicting the available bandwidth. It mainly relies on network parameters such as packet loss, delay, and jitter to estimate the current available bandwidth and then control the sending rate to avoid packet loss, delay, and jitter caused by network congestion. The core idea of ​​BBR is to detect the maximum bandwidth and minimum delay of the network by adjusting the sending amount, and control the sending rate according to the detection results. The maximum bandwidth is obtained by sending data that exceeds the network capacity. When the delay begins to increase after increasing the sending amount, it is the maximum bandwidth at this time; the minimum delay is calculated by sending data that is lower than the network capacity. When the delay does not decrease after reducing the sending amount, the RTT at this time is the minimum delay. PCC is a probe-based congestion control algorithm that does not need to accurately estimate the network bandwidth, but detects the network status by sending a group of probe packets, and then adjusts the transmission rate to achieve congestion control.

[0094] However, the existing technology adds some fields to the RTP (Real-time Transport Protocol) and RTCP (RTP Control Protocol) data packets of XR video content to achieve the simultaneous arrival of control information and media information. This only adjusts the transmission method or the transmission protocol itself, without considering the characteristics of XR video content.

[0095] Exemplarily, the XR video content that needs to be transmitted in the XR application scenario may include panoramic video, stereoscopic video, light field video, etc. Among them, panoramic video is a video shot in all directions at 360 degrees with a three-dimensional camera. When watching the video, the user can adjust the video up and down, left and right to watch it at will. Stereoscopic video is a new video technology that can provide a three-dimensional sense. Stereoscopic video generally has two video channels, and the amount of data is much larger than that of single-channel video. Light field is an advanced capture, stitching and rendering algorithm. By generating motion parallax and very realistic textures and lighting for static captured images, light field video can provide users with extremely high-quality presence. It can be seen that XR video content has the characteristics of huge transmission data volume. For example, the display essence of stereoscopic video is to quickly render a series of three-dimensional models at a speed of 30 frames per second, which allows the experiencer to freely explore the three-dimensional scene during video playback, which leads to a strong requirement for high bandwidth and low latency for XR video transmission. Since WebRTC itself is suitable for audio and video calls, it pays much more attention to latency than video quality. Therefore, it performs poorly in high-bandwidth scenarios and will give a lower bit rate to avoid the risk of network congestion during high-bitrate transmission. Video quality is also a very important part of XR applications. Even with the smoothest interactive operations, if the clarity of objects in the scene in the user's field of view is low, the user still cannot get a normal "immersive" experience.

[0096] Based on this, when using the congestion control method of WebRTC to implement transmission bit rate control in XR application scenarios, it is necessary to increase the bit rate increase multiplier in the congestion control mechanism of WebRTC. For example, if the congestion control mechanism of WebRTC defines that the bit rate will be increased by a value M under certain conditions, this embodiment stipulates that the bit rate will be increased by N times the value M under such conditions, thereby being more adaptable to high-bandwidth XR scenarios.

[0097] As another exemplary implementation, some bit rate adjustment rules may be pre-set, and the bit rate value matching the video quality index may be obtained by matching the determined video quality index with the bit rate adjustment rules. For example, the bit rate adjustment rules include bit rate values ​​corresponding to different video quality levels, such as bit rate A for the first level, bit rate B for the second level, and bit rate C for the third level, and a corresponding index interval is set for each video quality level. Based on this, after determining that the video quality index does not meet the preset index condition, the corresponding video quality level is first determined based on the index interval met by the video quality index, and then the bit rate value corresponding to the video quality level is obtained, so that the bit rate matching the video quality index can be obtained.

[0098] S540: Transmit the view to be transmitted to the client node based on the determined bit rate.

[0099] Based on the determined bit rate, the video to be transmitted is re-encoded and sent, and the view to be transmitted can be transmitted to the client according to the bit rate. Exemplarily, the server is configured with an encoder, a transmitter, and a retransmission controller for data transmission processing. By sending the determined bit rate to the encoder, the transmitter, and the retransmission controller respectively, the encoder, the transmitter, and the retransmission controller can be scheduled to perform corresponding video transmission processing according to the bit rate, so as to smoothly transmit the view to be transmitted to the client.

[0100] From this, it can be seen that the technical solution provided in this embodiment can ensure the video quality while ensuring the delay of the video to be transmitted by making relevant decisions on the XR video transmission bit rate, which is in line with the video transmission requirements of the XR scenario, and can therefore enable the experiencer to obtain a better user experience.

[0101] To further facilitate understanding of the core idea of ​​the technical solution provided in the embodiment of the present application, the technical solution provided in the embodiment of the present application is briefly introduced with an exemplary application scenario. The exemplary application scenario is specifically an XR application scenario for transmitting stereoscopic video.

[0102] See first Figure 6 , Figure 6 An architecture diagram of a stereoscopic video transmission system is illustrated. The stereoscopic video transmission system includes a client node 610 and a server node 620. The client node 610 includes a user input device, a PC, a smart phone, and an all-in-one device. The user input device is also an XR wearable device. The PC, the smart phone, and the all-in-one device are deployed with the WebRTC protocol. The server node 620 is also deployed with the WebRTC protocol to implement XR video related transmission between the client node 610 and the server node 620. The WebRTC protocol specifically adopts the GCC congestion control method to implement congestion control.

[0103] The server node 620 also has three functional modules deployed therein, namely, a transmission view candidate module, a view QoE value prediction module and a transmission bit rate decision module. The transmission view candidate module is used to predict candidate views matching the pose information according to a video acquisition request carrying pose information uploaded by the client node 610, using multiple preset view prediction models. The view QoE value prediction module is used to predict the QoE values ​​of each candidate view predicted by the transmission view candidate module, and perform fusion processing of the candidate views based on the predicted QoE values ​​to obtain the video to be transmitted. The transmission bit rate decision module is used to pre-encode the video to be transmitted to determine whether the video quality indicator corresponding to the video transmission at the preset bit rate meets the minimum quality condition. If it is determined that it does not meet the condition, a suitable bit rate is re-determined, and then the video to be transmitted is encoded according to the re-determined bit rate and transmitted to the client node 610.

[0104] It should be noted that the detailed functional implementation process of the transmission view candidate module, the view QoE value prediction module and the transmission bit rate decision module deployed in the server node 620 can be found in the contents recorded in the aforementioned embodiments, which will not be repeated here.

[0105] Figure 7 It is for Figure 6 The detailed flow chart of the stereoscopic video transmission system implementing the stereoscopic video transmission is shown in FIG. Figure 7 As shown, the server node 620 pre-stores a full-view stereoscopic view, and the video acquisition request carrying the position information is specifically sent to the server node 620 by the device worn by the experiencer. The transmission view candidate module deployed in the server node 620 first predicts the candidate views corresponding to multiple viewports, and then the view QoE value prediction module predicts the QoE value of each candidate view, and downsamples the corresponding candidate view based on the predicted QoE value, and then fuses the obtained downsampled image to obtain the video to be transmitted. At the same time, the transmission bit rate decision module will obtain the original target bit rate in the GCC congestion control module, and decide the bit rate suitable for transmitting the video to be transmitted, and then send this bit rate to the encoder, transmitter pacer and NACK retransmission controller respectively. It is easy to understand that the bit rate sent by the transmission bit rate decision module to the encoder is the target encoding bit rate, so that the encoder encodes the video to be transmitted according to the target encoding bit rate and obtains the corresponding video data packet; the bit rate sent by the transmission bit rate decision module to the transmitter is the target bit rate, so that the transmitter sends the video data packet according to the target bit rate; the target sent by the transmission bit rate decision module to the retransmission controller is the target retransmission bit rate, so that the retransmission controller transmits the retransmission data packet to the transmitter according to the target retransmission bit rate. It can be seen that the server node 620 uses the bit rate determined by the transmission bit rate decision module to allocate the bit rate, and schedules the bit rates of each part such as NACK retransmission, video encoding, and Pacer data packet transmission to achieve the optimization of transmission delay and video quality.

[0106] It is also worth mentioning that Figure 7 The corresponding processing flow in the client node 610 is also shown. The buffer buffer set in the client node 610 receives the video data packet transmitted by the transmitter, and then the decoder decodes the video data packet and transmits the decoded video data to the player for rendering and other processing, and finally transmits it to the wearable device worn by the experiencer for display. At the same time, the buffer also monitors the situation of receiving the video data packet itself. If it is detected that the transmitter has mis-sent or missed data packets, it sends an RTCP feedback message to the retransmission controller so that the retransmission controller performs retransmission processing of the corresponding data packet.

[0107] Figure 8 is a block diagram of an exemplary video transmission control device for augmented reality of the present application. The device can be configured in Figure 1 In the exemplary implementation environment, for example, it is specifically configured on the server device where the service end node 120 is located. Of course, the device can also be configured in other implementation environments, such as specifically configured on a terminal device or a server device in other implementation environments, and this embodiment does not limit this.

[0108] like Figure 8 As shown, the video transmission control device 800 applied to augmented reality includes:

[0109] The receiving module 810 is configured to receive a video acquisition request containing experiencer movement information sent by a client node; the prediction module 820 is configured to respectively predict candidate views matching the experiencer movement information through a plurality of preset view prediction models, wherein different view prediction models have different viewport positions after movement determined based on the experiencer movement information when predicting the candidate views, so that the predicted picture contents of each candidate view are correspondingly different; the acquisition module 830 is configured to obtain the experience quality value predicted for each candidate view, and perform fusion processing on the candidate views based on the experience quality values ​​corresponding to each candidate view to obtain a view to be transmitted; the transmission module 840 is configured to transmit the view to be transmitted to the client node.

[0110] In another exemplary embodiment, the acquisition module 830 includes:

[0111] A sorting unit is configured to sort each candidate view in an order from high to low or from low to high according to the experience quality value; a downsampling unit is configured to downsample the corresponding candidate views according to the sorting order of each candidate view to obtain a downsampled image corresponding to each candidate view; and a fusion unit is configured to fuse the downsampled images corresponding to each candidate view to obtain a view to be transmitted.

[0112] In another exemplary embodiment, the downsampling unit is configured to implement the following process:

[0113] Determine the corresponding downsampling frequency according to the sorting order of each candidate view, wherein the sorting order indicates that the candidate view with a higher experience quality value has a lower corresponding downsampling frequency; perform downsampling processing on the corresponding candidate view based on the determined downsampling frequency to obtain the downsampled image corresponding to each candidate view

[0114] In another exemplary embodiment, the prediction module 820 is configured to determine the viewport information after the movement according to the experiencer's movement information, and generate video content matching the viewport information based on the pre-stored full-view video to obtain the corresponding candidate views; accordingly, the fusion unit is configured to fuse the downsampled images corresponding to the candidate views to use the obtained fused image as the view to be transmitted.

[0115] In another exemplary embodiment, the prediction module 820 is configured to determine the viewport information after the movement according to the experiencer's movement information, and obtain the video blocks matching the determined viewport information, and use the video blocks as corresponding candidate views; accordingly, the fusion unit is configured to fuse the downsampled images corresponding to the candidate views to obtain a fused image, and obtain the full-view video after quality reduction processing, and use the fused image and the full-view video after quality reduction processing together as the view to be transmitted; wherein the picture quality of the video blocks is higher than the picture quality of the full-view video after quality reduction processing.

[0116] In another exemplary embodiment, the acquisition module 830 is further configured to input the historical experience quality values ​​of each candidate view into the experience quality prediction model, and obtain the experience quality values ​​predicted by the experience quality prediction model based on the historical experience quality values ​​of each candidate view. The historical experience quality value is determined based on at least one indicator of MTP delay, viewport offset distance, and video quality, and the MTP delay represents the length of time taken from the experiencer to move to the corresponding view screen display. Accordingly, the device also includes an update module, which is configured to obtain the real experience quality value fed back by the client node after transmitting the view to be transmitted to the client node, and update the experience quality prediction model based on the real experience quality value.

[0117] In another exemplary embodiment, the transmission module 840 includes:

[0118] A slicing unit is configured to slice the view to be transmitted to obtain a plurality of corresponding view slices; a pre-coding unit is configured to pre-code each view slice and determine a video quality index corresponding to the view to be transmitted based on the picture difference between the obtained coded image and the corresponding view slice; a decision unit is configured to determine a bit rate that matches the video quality index if the video quality index does not meet a preset index condition, and transmit the view to be transmitted to the client node based on the determined bit rate.

[0119] In another exemplary embodiment, the pre-encoding unit is configured to determine corresponding peak signal-to-noise ratio parameters and structural similarity parameters based on the picture difference between the encoded image and the corresponding view slice, and use the peak signal-to-noise ratio parameters and structural similarity parameters as video quality indicators.

[0120] In another exemplary embodiment, the decision unit is configured to send the determined bit rate to the encoder, the transmitter and the retransmission controller respectively, so as to schedule the encoder, the transmitter and the retransmission controller to perform corresponding video transmission processing according to the bit rate.

[0121] It should be noted that the video transmission control device for extended reality provided in the above embodiment and the video transmission control method for extended reality provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual applications, the video transmission control device for extended reality provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0122] In the video transmission control device for extended reality provided in the above embodiment, even if there is a sudden change in the movement trajectory of the experiencer or other factors, the video picture corresponding to the accurate viewport can be predicted, thereby transmitting the video to be transmitted to the client node for extended reality related display to obtain a higher user experience.

[0123] An embodiment of the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the video transmission control method for extended reality provided in the above-mentioned embodiments.

[0124] Fig. 9 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig. 9 The computer system 900 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0125] like Fig. 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903, such as executing the method described in the above embodiment. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, ROM 902 and RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0126] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom is installed into the storage section 908 as needed.

[0127] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 909, and / or installed from a removable medium 911. When the computer program is executed by a central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0128] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer program contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0129] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0131] Another aspect of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video transmission control method for augmented reality as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0132] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the video transmission control method for extended reality provided in each of the above embodiments.

[0133] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

[0134] It should also be noted that in the specific implementation of this application, data related to flow status, heartbeat information, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, the permission or consent of the experiencer is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A video transmission control method applied to augmented reality, characterized in that: include: Receive a video acquisition request including the experiencer's movement information sent by the client node; Predicting candidate views matching the movement information of the experiencer respectively by using a plurality of preset view prediction models, wherein different view prediction models predict different viewport positions after movement determined based on the movement information of the experiencer when predicting the candidate views, so that the picture contents of the predicted candidate views are correspondingly different; Acquire a predicted experience quality value for each candidate view, and perform fusion processing on the candidate views based on the experience quality values ​​corresponding to the candidate views to obtain a view to be transmitted; The view to be transmitted is transmitted to the client node.

2. The method according to claim 1, characterized in that: The obtaining of the predicted experience quality value for each candidate view, and performing fusion processing on the candidate views based on the experience quality values ​​corresponding to the candidate views to obtain the view to be transmitted includes: Sort the candidate views in order of high to low or low to high quality of experience values; Downsampling the corresponding candidate views according to the sorting order of the candidate views to obtain downsampled images corresponding to the candidate views; The downsampled images corresponding to the candidate views are fused to obtain the view to be transmitted.

3. The method according to claim 2, characterized in that The downsampling the corresponding candidate views based on the sorting order of the candidate views to obtain downsampled images corresponding to the candidate views includes: Determine the corresponding downsampling frequency according to the sorting order of each candidate view, wherein the sorting order indicates that the candidate view with a higher experience quality value has a lower corresponding downsampling frequency; Downsampling processing is performed on the corresponding candidate views based on the determined downsampling frequency to obtain downsampled images corresponding to each candidate view.

4. The method according to claim 2, characterized in that: The predicting the candidate views matching the movement information of the experiencer respectively by using the preset multiple view prediction models includes: Determine the viewport information after the movement according to the movement information of the experiencer; Video content matching the viewport information is generated based on the pre-stored full-view video to obtain corresponding candidate views.

5. The method according to claim 4, characterized in that The fusing downsampled images corresponding to the candidate views to obtain the view to be transmitted includes: The down-sampled images corresponding to the candidate views are fused, so as to obtain a fused image as the view to be transmitted.

6. The method according to claim 2, characterized in that The predicting the candidate views matching the movement information of the experiencer respectively by using the preset multiple view prediction models includes: Determine the viewport information after the movement according to the movement information of the experiencer; A video block matching the determined viewport information is obtained, and the video block is used as a corresponding candidate view.

7. The method according to claim 6, characterized in that The fusing downsampled images corresponding to the candidate views to obtain the view to be transmitted includes: The downsampled images corresponding to the candidate views are fused to obtain a fused image; A full-view video after quality reduction processing is obtained, and the fused image and the full-view video after quality reduction processing are used together as the view to be transmitted; wherein the picture quality of the video block is higher than the picture quality of the full-view video after quality reduction processing.

8. The method according to claim 1, characterized in that: The obtaining of the predicted experience quality value for each candidate view and generating a view to be transmitted based on the experience quality value corresponding to each candidate view includes: Inputting the historical experience quality value of each candidate view into the experience quality prediction model, wherein the historical experience quality value is determined based on at least one indicator of MTP delay, viewport offset distance, and video quality, and the MTP delay represents the length of time taken from the experiencer to move to the corresponding view screen display; The experience quality value predicted by the experience quality prediction model based on the historical experience quality value of each candidate view is obtained.

9. The method according to claim 8, characterized in that The method further comprises: After transmitting the view to be transmitted to the client node, obtaining a real experience quality value fed back by the client node; The experience quality prediction model is updated based on the actual experience quality value.

10. The method according to claim 1, characterized in that The transmitting the view to be transmitted to the client node includes: Slicing the view to be transmitted to obtain a plurality of corresponding view slices; Performing pre-coding processing on each view slice, and determining a video quality indicator corresponding to the view to be transmitted based on a picture difference between the obtained coded image and the corresponding view slice; If the video quality index does not meet the preset index condition, determining a bit rate that matches the video quality index; The to-be-transmitted view is transmitted to the client node based on the determined bit rate.

11. The method according to claim 10, characterized in that The pre-encoding processing is performed on each view slice, and based on the picture difference between the obtained encoded image and the corresponding view slice, the video quality indicator corresponding to the view to be transmitted is determined, including: Determining corresponding peak signal-to-noise ratio parameters and structural similarity parameters based on picture differences between the coded image and the corresponding view slice; The peak signal-to-noise ratio parameter and the structural similarity parameter are used as the video quality index.

12. The method according to claim 10, characterized in that The transmitting the view to be transmitted to the client node based on the determined bit rate includes: The determined bit rate is sent to the encoder, the transmitter and the retransmission controller respectively, so as to schedule the encoder, the transmitter and the retransmission controller to perform corresponding video transmission processing according to the bit rate.

13. A video transmission control device applied to augmented reality, characterized in that: include: A receiving module, configured to receive a video acquisition request including movement information of an experiencer sent by a client; A prediction module, configured to predict candidate views matching the experiencer's movement information respectively through a plurality of preset view prediction models, wherein different view prediction models have different viewport positions after movement determined based on the experiencer's movement information when predicting candidate views, so that the predicted picture contents of each candidate view are correspondingly different; An acquisition module configured to acquire a predicted experience quality value for each candidate view, and perform a fusion process on the candidate views based on the experience quality values ​​corresponding to the candidate views to obtain a view to be transmitted; The transmission module is configured to transmit the view to be transmitted to the client.

14. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the video transmission control method for extended reality as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the video transmission control method applied to augmented reality according to any one of claims 1 to 12.

Images