Video processing method and related device

Abstract

The invention discloses a video processing method and a related device, the method is applied to a video processing device, the device and a cloud application are arranged in a data center, a communication connection is established between the data center and a terminal device, and the method comprises the following steps: rendering to generate original video data, the original video data is obtained by rendering original model data corresponding to the original video data, the original model data is generated by the cloud application based on an application operation instruction sent by the terminal equipment through communication connection, and the original video data comprises M rendering layers; m rendering layers in the original video data are coded into i coding layers to generate a target video code stream, the first part of rendering layer in the M rendering layers is coded into the first coding layer,..., the ith part of rendering layer is coded into the ith coding layer, and i is greater than or equal to 1 and smaller than or equal to M. According to the method, layered coding is carried out on the rendering result with the visual attribute obtained based on layered rendering, the mutual influence between layers is reduced, and the controllability of the picture effect is enhanced.

Description

[0001] This application claims priority to Chinese patent application No. 202310893708.3, filed on July 19, 2023, and entitled “A video encoding method, device and system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of network technology, and in particular to a video processing method and related devices. Background Art

[0003] Layered coding, also known as scalable video coding (SVC), is used to encode video signals into layers and output multi-layer streams including a base layer and an enhancement layer. When bandwidth resources are insufficient, only the base layer stream is transmitted and decoded. When bandwidth resources are sufficient, the enhancement layer stream can be transmitted and decoded to improve the decoding quality of the video.

[0004] In the related technology, the layered coding encoder divides the video in terms of time, space, and quality, encodes the original video data obtained from the rendering end to obtain the basic layer and the enhancement layer, and sends the video stream including the basic layer and the enhancement layer to the terminal device according to the transmission capacity of the network. The terminal device is equipped with a decoding module to decode the video stream, reconstruct the video frame and display it.

[0005] However, due to the loss caused by data compression during rendering encoding and transmission, the layers in this video processing method will affect each other, resulting in the rendering content being unable to be intuitively controlled by the user, affecting the user experience. Summary of the invention

[0006] The present application provides a video processing method and related devices, which utilizes layered rendering information for layered encoding to reduce the mutual influence between image layers, allowing users to flexibly control image rendering results and maximize the benefits of layered rendering. The technical solution is as follows:

[0007] In a first aspect, a video processing method is provided, which is applied to a video processing device, wherein the video processing device and a cloud application are arranged in at least one data center, and a communication connection is established between the at least one data center and a terminal device, and the method comprises: rendering to generate original video data, wherein the original video data is rendered by original model data corresponding to the original video data, wherein the original model data is generated by the cloud application based on an application operation instruction sent by the terminal device through a communication connection, and the original video data comprises M rendering layers, where M≥2; encoding the M rendering layers in the original video data into i coding layers to generate a target video code stream, wherein a first part of the rendering layers in the M rendering layers is encoded as a first coding layer, a second part of the rendering layers in the M rendering layers is encoded as a second coding layer…an i-th part of the rendering layers in the M rendering layers is encoded as an i-th coding layer, where M≥i≥1; and sending the target video code stream to the terminal device.

[0008] It can be seen that in the process of rendering and encoding the original model data to generate a video code stream, the video processing device first renders the original model data into rendering data containing M rendering layers, and encodes some of the rendering layers into one encoding layer, and encodes a total of i encoding layers to complete layered encoding. Among them, some rendering layers are one or more rendering layers in the M rendering layers, but not all of the M rendering layers. Optionally, one of the M rendering layers can be rendered into one encoding layer, or multiple rendering layers in the M rendering layers can be rendered into one encoding layer. Thus, layered encoding is performed with the rendering layers obtained by layered rendering as the dimension, reducing the mutual influence between layers in the picture, avoiding the loss of overall details of the picture due to encoding and transmission, and the user's control over the picture can be directly reflected in the final rendered picture through rendering, encoding, and transmission, which enhances the controllability of the picture effect and improves the user experience.

[0009] According to a possible implementation manner of the first aspect, wherein the first partial rendering layer is the first rendering layer among M rendering layers, the second partial rendering layer is the second rendering layer among the M rendering layers…the i-th partial rendering layer is the i-th rendering layer among the M rendering layers, the method further includes: receiving a picture effect configuration instruction, where the picture effect configuration instruction is used to turn on / off an effect of a target rendering layer among the M rendering layers; rendering to generate original video data, including: starting / stopping rendering the target rendering layer according to the picture effect configuration instruction, so as to start / stop encoding a coding layer corresponding to the target rendering layer.

[0010] Specifically, the video processing device encodes a certain rendering layer among the M rendering layers into a certain coding layer in a one-to-one correspondence, and the picture effect configuration instruction received by the video processing device directly indicates the effect of turning on or off a certain rendering layer. Therefore, the adjustment instruction sent by the user for a certain picture layer can not only be directly executed in the rendering stage, but also be encoded as the coding layer corresponding to the picture layer or rendering layer in the coding stage, thereby avoiding the loss of overall picture details due to resource status fluctuations during the coding and transmission stages when adjusting the picture layer or rendering layer. For example, when a user turns on a certain picture layer, the picture layer is then rendered into the corresponding rendering layer, encoded into the corresponding coding layer and transmitted to the user, and the overall picture quality such as frame rate and resolution will not be affected, thereby enhancing the user's controllability of the picture. Optionally, the user can perform picture effect configuration operations on the picture effect configuration interface displayed on the terminal device, and then send a picture effect configuration instruction to the video processing device. Optionally, the screen effect adjustment instruction sent by the user can be displayed on the screen effect configuration interface as a selection box or drop-down box for turning on or off a certain screen layer, can be displayed as a selection box for clicking and dragging the effect level of a certain screen layer, can also be displayed as a selection box for adjusting the priority of different screen layers, etc. Optionally, the screen layer name displayed on the above screen effect configuration interface is different from the corresponding rendering layer name.

[0011] According to a possible implementation manner of the first aspect, the method further includes: receiving a picture effect configuration instruction, where the picture effect configuration instruction is used to enhance / weaken the effect of a target rendering layer among the M rendering layers; rendering to generate original video data includes: improving / reducing the rendering quality of the rendering target rendering layer according to the picture effect configuration instruction.

[0012] Specifically, the picture effect configuration instruction received by the video processing device indicates whether to increase or decrease the rendering quality of a certain rendering layer. In this way, when the video processing device receives the picture effect configuration instruction, it can adjust the rendering intensity of a certain rendering layer in the rendering stage according to the instruction. Moreover, since the subsequent encoding stage will encode based on the rendering layer as the dimension, the impact of the picture effect configuration instruction on the rendering layer can be directly reflected in the final presented picture through rendering, encoding, and transmission, thereby enhancing the controllability of the picture effect and improving the user experience.

[0013] According to a possible implementation of the first aspect, among the i coding layers, the first coding layer is the original base layer, and the other coding layers among the i coding layers are the original enhancement layers. The method further includes: obtaining resource status information, where the resource status information is used to indicate the status of one or more of the rendering pipeline, encoder, communication connection, and terminal device; determining, according to the resource status information, that the resource status can satisfy sending at least m original enhancement layers, where m ≤ i - 1; reconstructing the original base layer and m original enhancement layers into a target base layer, and the first bitstream includes the bitstream corresponding to the target base layer.

[0014] That is, on the basis of hierarchical coding, by combining the status information of one or more of the resources such as the rendering pipeline, encoder, communication connection, and terminal device, the coding layers are dynamically scaled by merging enhancement layers or splitting the base layer. For example, when the resource status can satisfy sending at least m enhancement layers, the m enhancement layers and the base layer are merged. Optionally, it can be determined according to the resource status information that the long-term stable resource status within a period of time can satisfy sending a certain number of enhancement layers, and these enhancement layers and the base layer are selected to be merged. Thus, while ensuring the complete transmission of the information of these coding layers, the bitrate consumption caused by the poor decorrelation between layers in hierarchical rendering and hierarchical coding itself is saved.

[0015] According to a possible implementation of the first aspect, after reconstructing the original base layer and m original enhancement layers into a target base layer, the method further includes: determining that the resource status cannot satisfy sending at least m original enhancement layers, and can satisfy sending at least n original enhancement layers, where n < m; reconstructing the original base layer and n original enhancement layers into a target base layer; stopping rendering the (n + 2)-th rendering layer, the (n + 3)-th rendering layer... the (m + 1)-th rendering layer.

[0016] That is, based on hierarchical coding, by combining the state information of one or more of the resources such as the rendering pipeline, the encoder, the communication connection, and the terminal device, the coding layer is dynamically scaled by merging enhancement layers or splitting the base layer. For example, after confirming that the resource state can satisfy sending multiple enhancement layers and merging these enhancement layers and the coding layer, that is, at this time, the base layer is already the base layer merged with the enhancement layers. If it is determined according to the resource state information at this time that the resource state can only satisfy sending fewer enhancement layers. For example, previously, m original enhancement layers (i.e., initial, unmerged enhancement layers) could be sent, and now only n enhancement layers can be sent, where n < m. Then the system reconstructs the original base layer and n enhancement layers into the target base layer. In other words, the new base layer that was previously merged with m enhancement layers is split into the above-mentioned target base layer and m - n layers. Optionally, it can be determined according to the resource state information that the long-term stable resource state within a period of time can satisfy sending a certain number of enhancement layers, and these enhancement layers and the base layer are selected to be merged. Thus, while ensuring the complete transmission of the information of these coding layers, the bitrate consumption caused by the poor decorrelation between layers in hierarchical rendering and hierarchical coding itself is saved.

[0017] According to a possible implementation manner of the first aspect, the method further includes: based on the picture effect configuration instruction, estimating the resource occupancy information and / or the picture effect information, where the resource occupancy information is used to describe the resource occupancy situation of the terminal device or the communication connection after encoding the original video data according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after encoding the original video data according to the picture effect configuration instruction; sending the resource occupancy information and / or the picture effect information to the terminal device.

[0018] That is, since the resource capabilities of rendering pipelines, encoders, communication connections, and terminal devices are relatively fixed, when the video processing device receives a picture effect configuration instruction, it can estimate the resource occupancy of the video processing system in the current resource state when the above picture effect configuration instruction is applied based on the configuration indicated by the instruction and various resource status information, such as CPU occupancy, GPU occupancy, memory occupancy, bandwidth occupancy, etc., and send these estimated information to the terminal device, and optionally display these estimated information on the resource configuration interface provided to the user. When the picture effect configuration instruction received by the video processing device indicates that when the picture effect configuration is greatly improved, the resource status is insufficient to support the effect configuration, and then, it will cause layered coding to lose some picture details in the short term by discarding some enhancement layers. For example, when the user sets a lower priority for the water wave effect layer in the rendering layer, then when other high-priority picture effect configurations are improved, resulting in an increase in the bit rate and the bandwidth being unable to support the sending of the low-priority enhancement layer, the water wave effect layer will be discarded first during the layered encoding and transmission process. At this time, the picture effect information can be described as a weakened water wave effect or a water wave effect turned off. Optionally, this estimated information is displayed on the resource configuration interface provided to the user.

[0019] In a second aspect, a video processing method is provided, which is applied to a video processing device, and the method includes: receiving a target video code stream, where the target video code stream is generated by encoding original video data, the original video data includes M rendering layers, the target video code stream includes at least one coding layer, the at least one coding layer includes a basic layer and / or at least one enhancement layer, and the i-th coding layer in the target video code stream is obtained by encoding part of the rendering layers in the M rendering layers, M≥2, M≥i≥1; decoding the i-th coding layer into the part of the rendering layer corresponding to the i-th coding layer, so as to decode the target video code stream.

[0020] Specifically, the video processing device is used to decode the target code stream, which is generated by layered rendering and layered coding, and a certain coding layer of the layered coding of the target code stream is obtained by encoding a part of the rendering layers in the rendering layers obtained by layered rendering, that is, the layered coding is encoded with the rendering layers obtained by layered rendering as the dimension. Furthermore, the video processing device is built with a decoder to decode the received coding layer into a corresponding partial rendering layer to decode the target video code stream. Optionally, the partial rendering layer may include one or more rendering layers of the M rendering layers, but not all rendering layers. Optionally, the video processing device is located in the user's terminal device.

[0021] According to a possible implementation of the second aspect, the method also includes: sending resource status information, the resource status information is used to determine that the resource status can satisfy the sending of at least m original enhancement layers, m≤M-1, the resource status information includes status information of the terminal device, the status information of the terminal device includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay.

[0022] Specifically, the video processing device collects and sends resource status information of the terminal device, and the resource status information includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay. Optionally, the video processing device sends the resource status information to another video processing device that performs encoding and rendering through a communication connection. Thus, the other video processing device can confirm that the resource status can meet the requirements of sending at least m original enhancement layers based on the resource status information.

[0023] In a third aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method in the first aspect. The video processing device includes at least one module, and the at least one module is used to implement the video processing method provided in the first aspect.

[0024] In a fourth aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method in the second aspect. The video processing device includes at least one module, and the at least one module is used to implement the video processing method provided in the second aspect.

[0025] In a fifth aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory;

[0026] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the video processing method described in the first aspect, or executes the video processing method described in the second aspect.

[0027] In a sixth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the video processing method described in the first aspect or the video processing method described in the second aspect.

[0028] In a seventh aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the video processing method described in the first aspect above, or executes the video processing method described in the second aspect above.

[0029] The technical effects obtained in the above-mentioned third to seventh aspects are similar to the technical effects obtained by the corresponding technical means in the first and second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0031] Figure 1 is a schematic diagram of the architecture of a video processing system provided in an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0033] Figure 3 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of the architecture of an encoding module provided in an embodiment of the present application;

[0035] Figure 5a-5e This is a schematic diagram of a cloud application configuration interface provided by an embodiment of the present application;

[0036] Figure 6a-6c This is a schematic diagram of a layered coding implementation method provided in an embodiment of the present application;

[0037] Figure 7a-7c It is a schematic diagram of the architecture of an encoding module provided in an embodiment of the present application;

[0038] Figure 8 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0039] Fig. 9 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0040] Fig.10 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0041] Fig.11 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0042] Fig.12 is a structural schematic diagram of a video processing device provided in an embodiment of the present application;

[0043] Fig.13 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0044] Fig.14 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0045] Fig.15 It is a schematic diagram of the network connection relationship of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] First, some terms and related technologies involved in the present invention are explained in conjunction with the drawings to facilitate understanding by those skilled in the art.

[0049] Layer rendering: Split the overall rendering process into several different layers, render each layer separately and save the corresponding rendering results. Layer rendering saves different screen elements or data in different layers, and then piece them together to form the final rendering effect. For example, it can be divided into foreground layer, mid-ground layer, and background layer according to the content of the rendering; for example, it can be divided into color layer, highlight layer, shadow layer, reflection layer, refraction layer, luminous layer, etc. according to the visual attributes of the rendering; for example, it can be layered according to visual prominence. Reasonable layered rendering can not only improve the speed, but also make it convenient to adjust various effects in the later stage.

[0050] Scalable video coding (SVC): Also known as scalable video coding or hierarchical video coding, it is scalable and hierarchical. It can be divided in terms of frame rate, resolution, and quality. One encoding can output multiple layers of code streams, which are divided into a basic layer and an enhanced layer, suitable for different terminals and network conditions. The basic layer occupies relatively less bandwidth resources and ensures basic video quality. The basic layer plus the enhanced layer can obtain a better frame rate, resolution or quality. The scalability of SVC coding is divided into temporal scalability, spatial scalability, and quality scalability. Temporal scalability refers to decomposing a video stream into information representing different frame rates, spatial scalability refers to decomposing a video stream into information representing different resolutions, and quality scalability refers to decomposing pixel values ​​into different levels.

[0051] Base layer: The lowest layer of data in layered coding. It contains basic information about the video, such as resolution, frame rate, color, etc., which enables the decoder to decode the main content of the video normally, but the video quality may be low. The data in the base layer can be decoded independently or combined with data from other enhancement layers to improve the quality, resolution or frame rate of the video. The data in the base layer usually occupies a lower bit rate and is suitable for transmission in networks with limited bandwidth or for playback on terminals with lower performance.

[0052] Enhancement layer: Data other than the lowest layer in layered coding. It contains additional information about the video, such as higher frame rate, resolution, quality, etc. It can be combined with the data of the base layer to improve the performance and experience of the video. The data of the enhancement layer cannot be decoded independently and needs to rely on the data of the base layer or other enhancement layers. The data of the enhancement layer can be dynamically adjusted according to different network conditions, terminal capabilities or user needs to achieve scalability and adaptability of the video. There can be multiple enhancement layer data, forming different levels, and each level can correspond to a dimension of time domain, spatial domain or quality.

[0053] Rendering pipeline: refers to the process of converting three-dimensional (3D) scene data into two-dimensional (2D) images. It is the core part of real-time graphics. The rendering pipeline usually consists of multiple stages, each of which processes the input data and passes the output to the next stage. The rendering pipeline generally includes the application stage, geometry stage, rasterization stage and fragment stage.

[0054] Before explaining the video encoding method provided by the embodiment of the present invention in detail, the application scenario of the embodiment of the present invention is first introduced.

[0055] Layered rendering technology is widely used in the production of games, film and television special effects, and animation. There are many forms of layered rendering. For example, it can be divided into foreground layer, midground layer, and background layer according to the content of the rendering; for example, it can be divided into color layer, highlight layer, shadow layer, reflection layer, refraction layer, luminous layer, etc. according to the visual attributes of the rendering. Layered coding refers to the division in time, space, or quality, and the output of a multi-layer code stream including a basic layer and an enhancement layer. The data of the basic layer can enable the decoder to decode the basic video content completely normally, thereby ensuring that the decoding end receives a smooth video image that can be viewed normally when the channel is limited or the channel environment is complex. When the channel environment is good or the channel resources are abundant, the enhancement layer data can be transmitted to improve the video quality.

[0056] Video coding is used to compress the original video, thereby reducing the size or bit rate of the video data without adversely affecting its quality under human visual perception. The coding service is built on cloud computing services or the manufacturer's physical servers. When the coding service is built on cloud computing services, it can save users the high investment of purchasing, building, and managing coding software and hardware, and avoid complex issues such as configuration optimization and coding parameter adaptation; at the same time, with the elastic scaling characteristics of cloud services, it can meet the actual coding business needs.

[0057] In the application scenarios of cloud games and digital humans, the main process from receiving instructions to layered rendering, layered encoding and transmission is as follows: the user sends signaling data from the terminal, such as game character control instructions, digital human action instructions, etc. The application generates model data according to the signaling data, sends it to the rendering device to render the model data, obtains the original video data, that is, generates the original video frame, and then uses the encoder to encode the original video data to obtain the encoded video code stream, and then transmits the video code stream to the user terminal device through the push streaming protocol, so that the decoding module installed in the user terminal device decodes the received video code stream and displays it through the user terminal device. In this process, rendering can be achieved through layered rendering, that is, the rendering device renders the original model data according to the color layer, highlight layer, shadow layer, reflection layer, refraction layer, luminous layer, etc.; encoding can be achieved through layered encoding, that is, the encoder performs layered encoding on the original video data in time, space or quality to obtain a basic layer and at least one enhancement layer, which is selectively transmitted to the user side according to the network status, and finally the terminal device 20 performs layered decoding to obtain the video frame and displays it.

[0058] However, when related technologies perform layered rendering and layered encoding on videos, layered encoding and network transmission affect the effect of layered rendering. Layered rendering technology enhances the possibility of later control and adjustment of different levels of effects. However, after layered encoding encodes a basic layer and at least one enhanced layer, due to factors such as network status, not all enhanced layers will be transmitted during transmission, which may cause loss of rendering content, thereby wasting computing resources consumed in the rendering process. When a certain layer of the picture fluctuates, it will affect the effects of other layers. In addition, when layered rendering is performed locally, users can intuitively adjust the picture according to visual attributes. For example, in local applications that provide fine picture control, users input commands to turn on highlight effects, and the highlight effects can be immediately presented in the picture without being affected by the encoding method or network status. However, when the rendering side is separated from the user side, such as when layered rendering occurs in the cloud or on the application manufacturer's server, the user-side experience cannot be consistent with that on the rendering side. That is, when the user wants to adjust the effect of a certain visual level based on intuitive visual attributes, the overall visual effect of the picture may be affected due to encoding or network fluctuations, making it difficult for the user to make subjective judgments and resulting in a poor experience.

[0059] Based on this, an embodiment of the present invention provides a video processing method, which performs layered encoding by utilizing layered rendered content, avoids mutual influence of different layers of content during whole frame encoding, matches rendering quality with encoding quality, reduces resource waste, and enhances image stability. Compared with traditional video processing solutions, the coupling between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of other contents. At the same time, the encoding layer can also be dynamically scaled according to the resource status of the current video processing system, saving resources and reducing information redundancy caused by layered rendering and layered encoding, which improves the quality of rendering and encoding while improving the user experience as a whole.

[0060] It should be understood that the video processing scenarios in the above examples are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. A person of ordinary skill in the art can appreciate that, with the emergence of new video processing scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0061] Please refer to Figure 1 , Figure 1 is a schematic diagram of a video processing system architecture provided by an embodiment of the present invention, such as Figure 1As shown, the video processing system includes a service-side device 10, a terminal device 20 and a communication connection 30, wherein the service-side device 10 includes a 3D application 101, a layered rendering module 102, an encoding module 103, and a control module 104, and the terminal device 20 includes a system and hardware 201, a control module 202, and a decoding module 203. The service-side device 10 and the terminal device 20 can communicate through the communication connection 30, so that the user can use cloud games, digital humans and other applications, control them in real time, and display real-time images on the terminal device 20.

[0062] In an embodiment provided by the present invention, the user 40 uses the 3D application 101 through an application installed on a terminal device 30 such as a personal computer or a smart phone, or uses the 3D application 101 through a web page. The application 101 may be a digital human application or a cloud game application, etc., which is not limited by the present invention. The 3D application 101 may be replaced by a 2D application, which is not limited by the present invention.

[0063] Next, taking the cloud game application service scenario as an example, the video processing method is explained. The service-side device 10, as the cloud game screen video transmitter, receives the operation instructions of the user 40, such as the game operation instructions for controlling the game character to make different actions or move on the scene, to generate the original model data, and transmit it to the rendering module 102 for layered rendering. Multiple rendering layers will be generated during the layered rendering process. Afterwards, the rendering layers rendered by the rendering module 102 are integrated into the original video data, which includes the frame-level pictures generated in real time by the rendering module 102, that is, the original video frames. The encoding module encodes the video frames generated by the rendering module into a base layer and an enhancement layer in units of frames. For example, when a spatially scalable layered encoding method is adopted, the encoding module 103 encodes the multiple original video frames to generate multiple images of different resolutions, wherein the base layer code stream encodes a low-resolution image, and the enhancement layer code stream is added on this basis, so that the decoding module 203 in the terminal device 20 can decode and obtain a high-resolution image. Thus, the original video data is encoded into a binary video code stream by the video encoder on the service side, and the video code stream is transmitted to the terminal device 20 via the communication connection 30. The terminal device 20 is equipped with a decoding module 203, which decodes the video code stream to obtain reconstructed video data, and the reconstructed video data includes multiple reconstructed video frames. Due to the layered encoding, when the network bandwidth fluctuates, different numbers and levels of coding layers are selected to be sent according to the needs and capabilities of the receiver, so that the receiver can decode video signals of different qualities or resolutions according to the received coding layers.

[0064] Since data compression is inevitable for the original model data and rendering data in the rendering, encoding, and transmission processes, and since the content is not perceived during encoding, the loss during data compression is completely determined by the encoder. In this process, the quality of different contents of the screen will affect each other due to direct content overlap and indirect bit rate competition. For example, if the text on the screen overlaps with the background, even if the text itself does not change between multiple frames, the text will become blurred if the complexity of the overlapping background texture increases or the motion intensity increases, which reduces the user experience.

[0065] In response to the above problems, an embodiment of the present invention proposes a video processing method, which uses layered rendering content for layered encoding to avoid mutual influence of different layers of content during whole frame encoding, so as to match the rendering quality with the encoding quality, reduce resource waste, and enhance the stability of the picture. Compared with traditional video processing solutions, the coupling degree between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of other contents. The encoding link becomes a function that can be controlled by application users and application developers. The rendering effect set by the application user can not only be fed back to the encoding link, but also penetrate into the rendering part, so that the rendering and encoding as well as the computing power of the network can be matched.

[0066] At the same time, when a video processing device based on layered rendering and layered coding processes video data, the following problems may occur: both layered rendering and layered coding have some drawbacks in decorrelation. Among them, layered coding requires higher coding complexity and bit rate, because layered coding requires different coding processing for each layer of information, such as quantization, transformation, entropy coding, etc., and it is also necessary to ensure synchronization and compatibility between the various layers; at the same time, layered coding also introduces additional redundant information, such as correlation and dependency between layers, and redundant coding within layers, which will lead to an increase in the bit rate after encoding. The layered rendered image itself has a strong correlation, such as the association between color and texture, or the influence between depth and illumination. Therefore, layered coding itself or layered rendering itself will waste a certain bit rate.

[0067] In view of the above problems, an embodiment of the present invention proposes a video processing method, in which the encoding module can maximize the bit rate saving when the resource status is stable by dynamically merging and decomposing the encoding layers when performing layered encoding on the original video data.

[0068] Please refer to Figure 2 , Figure 2 is a schematic diagram of a video processing system architecture provided by an embodiment of the present invention, such as Figure 2As shown, the video processing system includes a service-side device 10, a terminal device 20 and a communication connection 30, wherein the service-side device 10 includes a 3D application 101, a layered rendering module 102, an encoding module 103, and a control module 104, and the terminal device 20 includes a system and hardware 201, a control module 202, and a decoding module 203. The service-side device 10 and the terminal device 20 can communicate through the communication connection 30, so that the user can use cloud games, digital humans and other applications, control them in real time, and display real-time images on the terminal device 20.

[0069] In an embodiment provided by the present invention, the user 40 uses the 3D application 101 through an application installed on a terminal device 30 such as a personal computer or a smart phone, or uses the 3D application 101 through a web page. The application 101 may be a digital human application or a cloud game application, etc., which is not limited by the present invention. The 3D application 101 may be replaced by a 2D application, which is not limited by the present invention.

[0070] Next, taking the cloud game application service scenario as an example, the video processing method is explained. As the cloud game screen video transmitter, the service side device 10 receives the operation instructions of the user 40, such as the game operation instructions for controlling the game character to make different actions or move on the scene, to generate the original model data, and transmit it to the rendering module 102 for layered rendering. In this process, the service side device 10 configuration determines the rendering order and rendering quality of each rendering layer. The configuration can be preset by the system, such as the system presets the original template and makes the initial configuration according to the user's hardware and network status; it can also be manually configured by the user, such as the user sets to turn on or off a certain screen effect, adjust the quality level of a certain screen effect, or set the priority of different screen effects, and for example, the user selects a set of rendering configurations in the rendering template library provided by the application; it can also be that the service side adaptively adjusts the rendering configuration in real time according to information such as network status and hardware status, and the present invention does not limit this. Then, the rendering module 102 can adjust the specific rendering effect of a certain rendering layer according to the configuration sent by the user 40 or the system preset configuration. For example, in a scene manually configured by the user, when the user 40 inputs a picture control instruction to enhance the highlight effect, the rendering module 102 correspondingly increases the rendering intensity of the highlight layer.

[0071] Thus, the rendering module 102 renders the original video data to obtain multiple rendering layers, obtains the original video data, and sends it to the encoding module 103. The encoding module 103 encodes the rendering layers in the original video data into encoding layers one by one, that is, for example, the color layer is encoded as a basic layer, the highlight layer is encoded as an enhancement layer, the shadow layer is encoded as another enhancement layer, and the reflection layer is encoded as another enhancement layer... At this time, the enhancement layer encoded with the highlight layer is superimposed on the basic layer encoded by the color layer, so that the decoding module 203 in the terminal device 20 can decode and obtain a high-resolution image. Thus, the original video data is layered encoded by the video encoder on the service side using the information of layered rendering to generate a binary video code stream, which is transmitted to the terminal device 20 via the communication connection 30. The terminal device 20 is equipped with a decoding module 203 to decode the video code stream to obtain reconstructed video data, which includes multiple reconstructed video frames. The encoding module of this embodiment uses the information of layered rendering to perform layered encoding, so that when layered encoding sends encoding layers of different numbers and levels, the mutual influence between the layers is reduced, and the overall picture quality details of the video will not be lost in large quantities due to changes in a certain layer. At the same time, the rendering effect set by the user can be fed back to the encoding link and penetrate into the rendering part, and the rendering layer corresponding to the encoding layer can be controlled according to the subjectively perceptible picture configuration, thereby meeting the user's needs for controlling picture effects and quality in scenarios such as cloud games.

[0072] based on Figure 2 The video processing system architecture is combined with Figure 3 The video processing method provided by the embodiment of the present invention is introduced in detail by taking the cloud gaming application service scenario as an example.

[0073] See below Figure 3 , Figure 3 FIG. 1 is a flow chart of a video processing method provided by an embodiment of the present invention. Figure 3 As shown, the method includes but is not limited to the following steps:

[0074] S101 . The user 40 inputs an application operation instruction through the terminal device 20 .

[0075] The user 40 inputs the application operation instruction through the terminal device 20. For example, the user 40 is a user of the 3D application 101, that is, a cloud game player. The terminal device 20 is a device such as a personal computer, a smart phone, etc. The 3D application 101 can be an application installed on the terminal device 20, or its page can be accessed through the browser of the terminal device 20, and the present invention does not limit this. The application operation instruction can be used to control the movement of the game character, control the game character to release skills, or trigger the game cutscene animation, etc. The application operation instruction can be triggered by touching the touch screen, clicking the mouse, or inputting the keyboard, etc., and the present invention does not limit this.

[0076] S102. The terminal device 20 sends an application operation instruction.

[0077] Specifically, the terminal device 20 encapsulates the above-mentioned application operation instructions into a data message and sends it to the service-side device 10 through the communication connection 30. Specifically, the communication connection 30 can be the Internet, and the service-side device can be provided by a physical server or a data center. Furthermore, the above-mentioned physical server or data center can be one or more; it can also be provided by cloud computing resources such as virtual machines and containers; in other embodiments provided by the present invention, the service-side device 10 can refer to a collection of functional units implemented by software functions, for example, including codes running on multiple hosts / virtual machines / containers, and the present invention is not limited to this.

[0078] S103. The 3D application 101 generates original model data.

[0079] For example, the 3D application 101 is a cloud gaming application, which generates original model data according to the game operation instructions sent by the user 40. Optionally, the original model data may include information such as geometry, topology, color, texture, and animation. Optionally, in an embodiment provided by the present invention, the original model data may be scene graphics data or animation data, wherein the scene graphics data includes information such as objects, lighting, materials, and textures in the game or animation, and is used to generate visual effects in three-dimensional space. This data can be converted from vertex coordinates to pixels on the screen through a graphics rendering pipeline, and then the pixels are shaded by a shader, and finally output as a video. The animation data includes motion information of characters, objects, special effects, etc. in the game or animation, and is used to generate dynamic effects. This data can be bound together by skeletal animation, and then the posture of each frame is calculated by interpolation, and finally output as a video. The original model data can be stored in different formats, such as OBJ, FBX, GLTF, etc., and the present invention is not limited to this. In other embodiments provided by the present invention, 3D applications can be replaced by 2D applications, that is, the original model data can be two-dimensional or three-dimensional, and the present invention does not limit this. Specifically, when the instruction received by the cloud game is a game character control instruction, the original model data may include model data for the game character to perform corresponding actions and model data for background animations; when the instruction received by the cloud game is a cutscene trigger instruction, the original model data may include a fixed-duration cutscene model data, such as, including generating appearance model data based on the game user-defined character appearance, and combining skeletal drive data to generate the character's original model data, etc. The present invention does not limit the generation trigger conditions, generation methods, and content of the original model data.

[0080] S104. The 3D application 101 sends the original model data.

[0081] For example, the 3D application 101 sends the original model data to the rendering module 102 for further rendering to generate a video image that can be presented.

[0082] S105. The rendering module 102 renders the original model data into multiple rendering layers.

[0083] First, the rendering module 102 can layer the objects in the scene according to certain standards. Optionally, the objects can be divided into foreground layer, middle ground layer and background layer according to the rendered content. Specifically, if the 3D application 101 is a digital human application, the objects can be divided into skin layer, eye layer, mouth shape layer, expression layer, hair layer and accessories layer according to the rendered content. Figure 2 An embodiment of Figure 2 This is a schematic diagram of a layered rendering method provided by an embodiment of the present invention. As shown in the figure, the rendering module 102 divides the rendered content into a color layer, a highlight layer, a shadow layer, a reflection layer, a refraction layer, a luminous layer, etc. according to visual attributes. Then, independent rendering settings are performed for each rendering layer, including a renderer, rendering parameters, rendering channels, etc. Finally, each rendering layer is rendered to obtain a rendered image of each rendering layer.

[0084] The rendering module 102 may also be referred to as a rendering pipeline. Optionally, the rendering module 102 may include one or more of a ray tracing module, a rasterization module, and a shader module; it may also include one or more of a vertex processing module, a texture mapping module, an ambient occlusion rendering module, a high dynamic range rendering module, and a post-processing module; specifically, the rendering module may also include one or more of rendering modules with specific functions such as a particle rendering module, a lighting rendering module, a shadow rendering module, and a water wave rendering module, and the present invention is not limited to this.

[0085] S106. The encoding module 103 encodes the multiple rendering layers into multiple encoding layers to generate a video code stream.

[0086] The encoding module 103 encodes the multiple rendering layers generated by the rendering module 102 in step S105 into multiple encoding layers. Specifically, the encoding module 103 encodes one or more rendering layers into one encoding layer to generate a video bitstream containing multiple encoding layers. For example, when the rendering layer is divided into a color layer, a highlight layer, a shadow layer, a reflection layer, a refraction layer, and a luminous layer, the encoding module 103 can encode the color layer as a basic layer, the highlight layer as a first enhancement layer, and the shadow layer as a second enhancement layer... Optionally, the encoding module 103 can also encode the color layer and the shadow layer as a basic layer, encode the highlight layer as a first enhancement layer, and encode the reflection layer and the refraction layer as a second enhancement layer. For another example, when the game is a soldier battle game and the rendered screen is soldiers fighting on the grass, the rendering layer can be divided into a character layer (including soldiers), a background layer (including grass), and a text layer (including battle broadcasts). The encoding module 103 can encode the character layer as a first basic layer and the background layer as a second basic layer, so that the user end can present the characters and the background to ensure a relatively complete game screen, and so on.

[0087] An embodiment of the encoding module 103 for encoding multiple rendering layers into multiple encoding layers may be combined with Figure 4 , Figure 4 FIG. 1 is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. Figure 4 As shown in the figure, the result after the highlight layer rendering is sent to the encoder to be encoded as the base layer, and the result of the shadow layer to the reflection layer rendering is sent to the encoder to be encoded as the enhancement layer. Taking the shadow layer as an example, the encoder subtracts the reconstruction value of the previous layer encoder from the rendered video frame of the current layer to obtain the residual, and encodes the residual as the enhancement layer. The subsequent layers cyclically execute this process to obtain multiple enhancement layers.

[0088] S107. The encoding module 103 sends a video code stream.

[0089] Specifically, the video code stream includes multiple coding layers generated by the coding module 103 in step S106.

[0090] S108. The user 40 inputs a picture effect configuration instruction through the terminal device 20 to turn on / off the effect of the target rendering layer.

[0091] Specifically, the user 40 enters the screen effect configuration instruction on the interface provided by the cloud application 101 through the client installed on the terminal device 20 or the browser capable of accessing the cloud application via the communication connection 30 to turn on or off the effect of the target rendering layer. Figure 5a to Figure 5d , Figure 5a to Figure 5d Schematic diagram of a screen effect configuration interface provided by an embodiment of the present invention. Figure 5a to Figure 5d As shown, in Figure 5aIn the screen effect configuration interface, items that can be configured by the user include turning on or off a certain layer of screen and the effect level of a certain layer of screen, where the effect level can be divided into very low, low, medium, high, very high, etc., and the present invention does not limit this.

[0092] In other embodiments provided by the present invention, Figure 5b , the items that users can configure on the screen effect configuration interface are the screen layer priority preferences, such as highlight priority, shadow priority, etc. Figure 5c In the embodiment of the present invention, the screen layers displayed in the screen effect configuration interface can be divided according to the screen content, such as player characters, non-player characters, water surface effects, particle effects, ground and light, etc. Figure 5d , the cloud application 101 may be a digital human application, and the screen layers displayed in the screen effect configuration interface may be divided into skin, eyes, mouth shape, facial expression, hair and accessories according to the screen content. Thus, the user may turn on or off the corresponding rendering layer by clicking on or off, adjusting the level, setting the priority, etc., to configure the desired screen effect, and the present invention does not limit this.

[0093] Optionally, the video processing method and the video processing device provided by the embodiments of the present invention provide the user with a picture effect configuration interface and a picture effect configuration interface in the form of an end-side application or a browser, etc. The picture effect configuration interface is used to receive a picture effect configuration instruction sent by the user, so that the service side modifies the rendering configuration and the encoding configuration based on the picture configuration instruction. Optionally, when the occupancy rate of the hardware resources on the user side, such as the CPU, graphics card, etc., is close to saturation, or the bandwidth bit rate difference of the network is less than the threshold preset by the system or set by the user, the user attempts to send a picture effect configuration instruction to enhance the rendering quality of the highlight layer, but the system resources are insufficient to support the enhanced quality. The system estimates the resource occupancy or actual picture effect after enhancing the highlight layer rendering quality based on the instruction, and displays it in real time on the user's configuration interface for the user to determine whether to apply the configuration.

[0094] In another embodiment provided by the present invention, the developer of the application 101 can use the method provided by the embodiment of the present invention when designing the application. Figure 2 Video processing systems and Figure 3 The video processing method assists application design, for example, an example of an embodiment of the present invention can be referred to Figure 5e , Figure 5eIt is a schematic diagram of a design interface that can be provided to application developers. For example, when the application 101 is a cloud game, the developer of the cloud game can set different rendering coding priorities and rendering coding configurations for different layers of content during design to ensure the quality of high-priority content; at the same time, the developer can adjust the complexity and priority of rendering through real-time debugging. For example, if the background, character, special effect, and text are layered, if the developer finds that the background grass rendering quality is low, since the video picture presented by the video processing method in the embodiment of the present invention has the characteristics of end-to-end intuitive feedback, that is, it indicates that the texture complexity of the current background layer itself is high, or the rendering configuration or priority corresponding to the background layer is low, the developer can specifically reduce the texture complexity of the picture itself, or adjust the rendering configuration of the background layer, avoiding the situation in the related technology that all layers of the picture affect each other in the encoding and transmission stages, so that the problem cannot be intuitively located; the control module 104 can count the data of different layers and different contents (such as bit rate, encoding complexity, encoding quality), which can be used as a reference for developers in cloud game design to help developers better analyze problems. Optionally, the developer may provide an initial configuration or an optimal configuration. For example, the developer may set the color layer as the basic layer and the texture layer as the first enhancement layer. This configuration cannot be manually changed by the game player. In another example, the developer may provide an initial priority for each rendering layer for the game player's reference. For example, a recommended configuration prompt may be given when the player first enters the game, and the player may be guided to apply the configuration with one click. The embodiment of the present invention does not limit the method for providing the recommended configuration.

[0095] S109. The terminal device 20 sends a screen effect configuration instruction.

[0096] After receiving the configuration instruction from the user in step S108, the terminal device 20 sends a screen effect configuration instruction to the service side device, and the screen effect configuration instruction is used to instruct the rendering module to render according to the configuration, and specifically, it can be sent to the rendering module 102. In other embodiments provided by the present invention, the terminal device 20 can first send the configuration information generated according to the effect configuration instruction in step S108 to the control module 104 of the service side device 10, and then the control module 104 generates the screen effect configuration instruction and sends it to the rendering module 102, and the present invention is not limited to this.

[0097] Optionally, in other embodiments provided by the present invention, the screen effect configuration instruction can be generated by the control module according to resource status information. For example, the service side device 10 can generate a recommended initial screen effect configuration for the user 40 based on resource status information such as the processor configuration of the terminal device 20, the bandwidth size and stability of the communication connection 30, or the rendering and encoding capabilities of the service side device 10, or generate a recommended real-time screen effect configuration during the application running process, and send it to the rendering module 102. The present invention is not limited to this.

[0098] S110. The rendering module 102 starts / stops rendering the target rendering layer according to the picture effect configuration instruction.

[0099] Specifically, after receiving the picture effect configuration instruction, the rendering module 102 starts rendering according to the picture effect configuration. For example, if the picture effect configuration instruction indicates to start rendering of the highlight layer, the rendering module starts to use the renderer, rendering parameters and rendering channels related to the highlight layer to render and generate the highlight layer of the picture.

[0100] S111. The rendering module 102 starts / stops sending the target rendering layer.

[0101] For example, if the rendering module 102 receives a picture effect configuration instruction to render and generate a highlight layer, the highlight layer is sent to the encoding module 103 for the next step of encoding; if the rendering module 102 receives a picture effect configuration instruction to stop generating the highlight layer, it stops sending the highlight layer to the encoding module 103.

[0102] S112. The encoding module 103 starts / stops encoding the encoding layer corresponding to the target rendering layer.

[0103] Specifically, if the rendering module 102 starts to render the highlight layer and sends the highlight layer to the encoding module 103, the encoding module 103 starts to encode the encoding layer corresponding to the highlight layer. For example, if the encoding method of the encoding module is to encode a rendering layer into a corresponding encoding layer, the highlight layer is encoded into a basic layer or an enhancement layer; if the encoding method of the encoding module is to encode multiple rendering layers into corresponding encoding layers, the highlight layer and other rendering layers are encoded into a basic layer or an enhancement layer.

[0104] S113. Collect resource status information.

[0105] The control module 104 collects resource status information. Specifically, the control module 104 can collect the resource status information from the following sources: the encoder in the encoding module 103 in the server device 10, the rendering pipeline in the rendering module 102, the transmission network corresponding to the communication connection 30, and the terminal device 20, etc. An architecture for the control module 104 to collect resource status information from these modules can be referred to Figure 8 Provided embodiments. For example, the resource status information may include information such as the encoding bit rate, encoding complexity, encoding content scene, motion intensity, and layered encoding overhead in the encoder, and may also include information such as the rendering content complexity, rendering content changes, and 3D application rendering methods in the rendering pipeline, and may also include information such as the network real-time bandwidth, network fluctuations, packet loss rate, and bandwidth bit rate difference in the transmission network, and may also include information such as the CPU occupancy rate, layered decoding overhead, and layered decoding delay in the terminal device. That is, the resource status information is information reflecting the operating status of the video processing system including the user-side and service-side devices. It is worth noting that in other embodiments provided by the present invention, the reference resource status information for the control module to configure the layered rendering and layered encoding strategies includes one or more of the information in the above examples, and other indicators reflecting the software and hardware status and network status of the rendering and encoding system may also be used as a reference, and the present invention does not limit this.

[0106] S113a. Send terminal device resource status information.

[0107] Specifically, the terminal device 20 sends the terminal device resource status information to the control module 104 in the server-side device 10. For example, it may include CPU occupancy, layered decoding overhead, layered decoding delay, and the like.

[0108] S113b. Send rendering pipeline resource status information.

[0109] Specifically, the rendering module 102 sends the rendering pipeline resource status information to the control module 104 in the server-side device 10. For example, the rendering content complexity, rendering content change, 3D application rendering mode, and the like may be included.

[0110] S113c. Send transmission network resource status information.

[0111] Specifically, the communication connection 30 sends the transmission network resource status information to the control module 104 in the service side device 10. For example, it may include network real-time bandwidth, network fluctuation, packet loss rate, bandwidth bit rate difference, and so on.

[0112] S113d. Send encoder resource status information.

[0113] Specifically, the encoding module 103 sends encoder resource status information to the control module 104 in the server device 10. For example, the information may include encoding bit rate, encoding complexity, encoding content scene, motion intensity, layered encoding overhead, and the like.

[0114] S114. According to the resource status information, determine that the resource status can satisfy the requirement of sending at least m enhancement layers.

[0115] Specifically, the control module 104 determines, based on the collected resource status information, that the current resource status of the video processing system can satisfy the requirement of sending at least m enhancement layers.

[0116] For example, it can be judged based on the bit rate or bandwidth bit rate difference. When the bit rate fluctuates slightly, it can be judged that at least m enhancement layers can be sent for the next step of merging coding layers; when the bit rate fluctuates greatly, it can be judged that the system cannot stably send m enhancement layers, and the enhancement layers cannot be merged in the next step. The same is true when using the bandwidth bit rate difference for judgment. The bandwidth bit rate difference is the difference between the bandwidth and the bit rate. If the bandwidth continues to be greater than the bit rate over a period of time, it can be judged that at least m enhancement layers can be sent for the next step of merging coding layers.

[0117] For another example, it can be judged based on the encoding complexity. When the encoding complexity is high, it is judged that more resources will be consumed, and the number of enhancement layers that can be sent in the next step becomes smaller; for another example, it can be judged based on the content scenario. When the currently running application 101 is a racing game, because the racing game screen is rich in details and changes rapidly, the bit rate is relatively high, and accordingly, the number of enhancement layers that can be sent is relatively small; when the currently running application 101 is a document editing software, because the document editing software has more pure colors and less changes, the bit rate is relatively low, and accordingly, the number of enhancement layers that can be sent is relatively large; for another example, it can be judged based on the complexity of rendering content. For example, when the user-side configuration requires that its frame rate is stable at 90fps, and the rendering capability of the service-side device is constant, if the rendering complexity is high, the resource status may not be sufficient to support the rendering of m+2 enhancement layers, but can only support the rendering of m enhancement layers. Then the control module 104 determines that the resource status of the current video processing system can satisfy the sending of at least m enhancement layers.

[0118] In summary, the control module 104 determines, based on the collected resource status information, that the current resource capability of each module in the system is sufficient to support the sending of at least m enhancement layers. Optionally, the control module 104 determines, by counting changes in the resource status information within a certain period of time, that the current resource capability of each module in the system is sufficient to support the sending of at least m enhancement layers within a long stable period of time.

[0119] S115. Send a coding layer merge instruction.

[0120] Specifically, the control module 104 sends a coding layer merging instruction to the coding module 103 to instruct the coding module 103 to merge the original base layer and the m enhancement layers. In other embodiments provided by the present invention, when the control module 104 determines in step S114 according to the resource status information that the current resource status cannot satisfy the requirement of sending at least m enhancement layers but can only satisfy the requirement of sending at least n enhancement layers, the control module 104 may send a coding layer decomposition instruction in step S115 to instruct the coding module to decompose the merged base layer into a new base layer and mn enhancement layers.

[0121] S116. Reconstruct the original base layer and the m enhancement layers into a target base layer.

[0122] Specifically, after the encoding module 103 receives the encoding layer merging instruction sent by the control module 104, the original base layer and m enhancement layers are reconstructed into the target base layer, and this is used as the base layer for layered encoding, and the remaining encoding layers continue to be used as enhancement layers. The merged base layer reduces information redundancy compared to the multiple encoding layers before the merge, so that in the long-term stable state, the encoder only sends one necessary base layer and a small number of enhancement layers, reducing the information entropy loss caused by layered rendering and layered encoding, and saving hardware and network resources.

[0123] An embodiment for reconstructing an original base layer and multiple enhancement layers into a new base layer may be combined Figures 6a to 6c , Figures 6a to 6c FIG. 1 is a schematic diagram of a method for reconstructing an original base layer and multiple enhancement layers provided by an embodiment of the present invention. As shown in the figure, the steps of layered coding merging are as follows: First, Figure 6a , select layer m, layer m+1, and layer m+2 as the base layer and multiple enhancement layers to be merged; then, Figure 6b , use the reconstruction value of the last enhancement layer, i.e. layer m+2, to refresh the reconstruction value of the base layer; finally, as Figure 6c , the base layer is moved to the position of the last enhancement layer, i.e., layer m+2, and the rendering result corresponding to the layer is encoded as the base layer, and then a new video frame is encoded according to the new reconstruction value. Optionally, the decoding process is similar to the encoding process, except that the merging and decomposition are only performed at the encoder level, without involving the rendering part, and the decomposition process directly moves the base layer forward gradually.

[0124] An embodiment of the combined encoding module 103 for implementing layered encoding may be combined with Figure 7a , Figure 7a FIG. 1 is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. Figure 7aAs shown, specifically, in this embodiment, for example, the base layer is the color layer, and the enhancement layers are the texture layer, the highlight layer, and the shadow layer. The rendered color layer can be encoded using a general encoder in the relevant technology, and the rendered texture layer minus the color layer is the residual between the two layers. The color layer is added with the residual, and it can be encoded into the first enhancement layer in the coding layer. The encoder of the first enhancement layer is a private encoder. That is, it is implemented using a non-fully compliant video encoder and a private encoder, wherein the non-fully compliant video encoder needs to update the reconstruction value, and the private encoder is used to compress the residual. Optionally, the residual can be compressed in a lossless manner, or it can be compressed using the image compression method in the relevant technology.

[0125] When the control module 104 determines based on the resource status information that the color layer, texture layer and highlight layer are to be merged in the layered coding, the enhancement layer corresponding to the highlight layer needs to be used to update the basic layer reconstruction value during the coding process as the basic layer reference value in the coding process, so as to serve as the reference value when encoding the next frame.

[0126] Another embodiment of the combined encoding module 103 for implementing layered encoding can be combined with Figure 7b , Figure 7b FIG. 1 is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. Figure 7b As shown, specifically, in this embodiment, when it is necessary to merge the color layer and the texture layer, the merged frame directly refers to the base layer of the previous frame as the base layer reference value, that is, the coding layer corresponding to the highlight layer of the previous layer is directly used as the base layer reference value for inter-frame prediction. This encoding method is relatively simple to implement and can be implemented using a standard video encoder and a private encoder.

[0127] Another embodiment of the combined encoding module 103 for implementing layered encoding can be combined with Figure 7c , Figure 7c FIG. 1 is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. Figure 7c As shown, specifically, in this embodiment, the rendered color layer, texture layer, highlight layer and shadow layer are all encoded using a video encoder, that is, each rendering layer is regarded as a video frame for encoding, the color layer is the first frame, the texture layer is the second frame, and the highlight layer is the third frame. Therefore, it can be implemented based on a fully compliant video encoder and private outer layer scheduling, the enhancement layer encoder is consistent with the previous layer encoder, and when the control module 104 determines to merge multiple coding layers, the enhancement layer directly uses the previous layer to encode the video frame of the next layer, and uses the encoded enhancement layer code stream to refresh the basic layer reference value, wherein the encoding end can set the motion vector to 0 and fixed block division to accelerate the encoding process.

[0128] S117. Send video code stream.

[0129] Specifically, the encoding module 103 sends the encoded video code stream to the terminal device 20 on the user side through the communication connection 30, and the video code stream includes the merged coding layer. Optionally, after the video code stream is decoded by the decoder of the terminal device 20, the video picture can be displayed on the display device included in the terminal 20 device.

[0130] Please refer to Fig. 9 , Fig. 9 is a flow chart of a video processing method provided by an embodiment of the present invention, such as Fig. 9 As shown, in an embodiment provided by the present invention, the control module may include a short-term layered coding control module 1041 and a long-term layered coding control module 1042 , and the coding module 103 may include a short-term layered coding module 1031 and a long-term layered coding module 1032 .

[0131] The short-term layered coding module 1031 is used to control the frame-level coding data transmission and resource status information statistics. First, it receives the multi-layer code stream input by the coding module 103 and sends the basic layer to the transmission layer. Then, combined with the real-time or statistical resource status information as described above, it determines whether to send the next enhancement layer. This step is repeated until the enhancement layer cannot be sent. At the same time, optionally, real-time information such as the number of enhancement layers sent, the bandwidth bit rate difference, and the encoder coding complexity are counted, and the fluctuation coefficient is calculated based on this information and sent to the long-term layered coding module 1032 for reference.

[0132] The long-term layered coding module 1032 is used to control the generation and allocation of the enhancement layer and the base layer. When the long-term bandwidth can stably support the base layer and one or more enhancement layers, the module controlling the layering in the long-term layered coding module 1032 merges these base layers and enhancement layers into a new base layer, thereby increasing the coding efficiency and ensuring the coding stability; when the long-term bandwidth cannot stably support or the fluctuation coefficient is large, the module controlling the layering disassembles the base layer into a new base layer and enhancement layer until the base layer can meet the requirements. At the same time, optionally, for the enhancement layers and rendering layers that will not be used, the long-term layered coding module 1032 can stop the rendering pipeline in advance and control the number of enhancement layers, thereby saving rendering computing power.

[0133] In an embodiment provided by the present invention, the long-term layered coding module 1032 can accept a predefined template configured by a developer of the application 101. For example, the predefined template can configure information such as the layered rendering method of the application content, and based on this information, the entire system can be controlled to generate the most suitable initial rendering and coding configuration. At the same time, optionally, the long-term layered coding module 1032 can count the long-term stability information of the system over a period of time, such as the bit rate, computing power and other resource overheads required for different enhancement layers, and adjust through an adaptive control module to achieve the best overall effect. And this information can also be provided to the user side for reference, so that the user can participate in the adjustment of the system to achieve a better subjective experience.

[0134] Fig. 9 The embodiment of the present invention shows a flow chart of a method in which the control module 104 instructs the encoding module 103 to merge and decompose the encoding layers. The video processing method flow chart includes the following steps:

[0135] S201. Send the base layer.

[0136] Specifically, the short-term layered coding module 1031 in the coding module 103 sends the base layer to the terminal device 20 .

[0137] S202. Confirm that the resource status satisfies the transmission of the first enhancement layer, the second enhancement layer, and the third enhancement layer.

[0138] Specifically, the short-term layered coding control module 1041 in the control module 104 confirms that three enhancement layers can be sent currently through real-time resource information such as network bandwidth, and informs the short-term layered coding module 1031 .

[0139] S203. Send the first enhancement layer, the second enhancement layer, and the third enhancement layer.

[0140] Specifically, the short-term layered coding control module 1041 in the control module 104 instructs the short-term layered coding module 1031 to send the first enhancement layer, the second enhancement layer and the third enhancement layer.

[0141] S204. Confirm that the long-term resource status can satisfy the transmission of the first enhancement layer and the second enhancement layer.

[0142] Specifically, the long-term layered coding control module 1042 in the control module 104 determines that the long-term resource status can meet the transmission of the first enhancement layer and the second enhancement layer by counting the long-term stable resource status information of each module of the system, and informs the long-term layered coding module 1032. For example, the resource status at this time may fluctuate, and the lowest value of the fluctuation can meet the transmission of the first enhancement layer and the second enhancement layer, but cannot meet the long-term stable transmission of the third enhancement layer. Optionally, the third enhancement layer is adjusted in real time according to the network bandwidth status through the short-term layered coding control module to determine whether it can be sent.

[0143] S205. Combine the base layer, the first enhancement layer and the second enhancement layer into a new base layer a.

[0144] Specifically, the long-term layered coding control module 1042 instructs the long-term layered coding module 1032 to merge the base layer and the first enhancement layer and the second enhancement layer into a new base layer a. An embodiment of the merging can refer to Figure 6a , Figure 6b , Figure 6c and Figure 7a , Figure 7b , Figure 7c .

[0145] S206. Send the base layer a and the third enhancement layer.

[0146] Specifically, the long-term layered coding module 1032 sends the merged base layer a and the third enhancement layer to the terminal device 20.

[0147] S207. Confirm that the long-term resource status cannot satisfy the transmission of the second enhancement layer.

[0148] Specifically, the long-term layered coding control module 1032 confirms that the long-term resource state cannot meet the transmission of the second enhancement layer, and informs the long-term layered coding module 1032. That is, the long-term layered coding control module 1032 confirms that the long-term resource state of the system cannot meet the transmission of the second enhancement layer through the long-term stable resource state information in the system for a period of time.

[0149] S208. Decompose the base layer a into the base layer b and the second enhancement layer.

[0150] Specifically, the long-term layered coding control module 1032 decomposes the base layer a into the base layer b and the second enhancement layer. Optionally, the original base layer and enhancement layer may be reconstructed into the base layer b and the second enhancement layer.

[0151] S209. Confirm that the resource status cannot satisfy the transmission of the third enhancement layer.

[0152] Specifically, the short-term layered coding control module 1041 in the control module 104 confirms that the resource status cannot satisfy the transmission of the third enhancement layer, and informs the short-term layered coding module 1031 in the coding module 103 .

[0153] S210. Send the base layer b and the second enhancement layer.

[0154] Specifically, the short-term layered coding module 1031 in the coding module 103 sends the base layer b and the second enhancement layer to the terminal device 20 .

[0155] Please refer to Fig.10 , Fig.10 is an architecture diagram of a video processing system provided by an embodiment of the present invention, such as Fig.10 As shown, in an embodiment provided by the present invention, the video processing system is implemented in an edge-cloud collaborative manner, and the video processing system includes a service-side device 10a and a service-side device 10b, that is, Figure 2 The service side device 10 in the video processing system shown includes a service side device 10a and a service side device 10b, wherein the service side device 10a is provided by one or more cloud data centers, and the service side device 10b is provided by one or more edge cloud devices. The edge cloud is a technology that deploys cloud computing capabilities closer to data sources or users. The difference between it and the public cloud mainly lies in the deployment location, service objects, resource management and application scenarios. It can provide services nearby at the edge node, reduce network delays and jitters, and improve the response speed and quality of services. It can also filter, compress, aggregate and other processes on the data at the edge node, reduce the amount of data transmission, save network bandwidth and traffic, and reduce network costs.

[0156] In a scenario provided by an embodiment of the present invention, application 101 is a role-playing game, and the rendering layers of the game include a background layer, a character layer, a special effects layer, a reflection layer, an icon layer, and a text layer. Specifically, the background layer can be the background in which the characters are fighting, such as grass, a beach, etc., the special effects layer can be the special effects when the characters use skills in the battle, the reflection layer can be the reflection effect of objects such as characters and weapons, which is used to make the objects look more textured and realistic, and the icon layer and the text layer can be used to display game prompts, character names, player dialogues, etc.

[0157] In this embodiment, the original model data for rendering generated by the application can be distributed to the service-side device 10a and the service-side device 10b for rendering. For example, the background layer, character layer, special effects layer and reflection layer can be rendered, encoded and transmitted by the service-side device 10a, and the icon layer and text layer can be rendered, encoded and transmitted by the service-side device 10b. Since the contents in the icon layer and the text layer are mostly high-frequency information, the complexity of their rendering is lower than the complexity of encoding and transmission, that is, the required computing power is lower, but the required bandwidth is higher. Therefore, this part of information is rendered and encoded on the edge device through edge-cloud collaboration, and with the help of the better network quality and lower transmission bandwidth of the edge device, this information can be encoded with higher quality, which greatly improves the picture quality of some content on the terminal side and solves the pain point of poor quality of information such as text.

[0158] Please refer to Fig.11 , Fig.11 is an architecture diagram of a video processing system provided by an embodiment of the present invention, such as Fig.11 As shown, in one embodiment provided by the present invention, the cloud application 101 is specifically a digital human application 101, which can generate a virtual image that simulates human appearance, expression, voice and language with artificial intelligence (AI) reasoning technology, and can perform multimodal interaction and services in different scenes. In this embodiment, the digital human application 101 generates raw data such as mouth shape, face, and eyes with voice-driven, key point generation and other technologies. The raw data may include the dynamic effect and position information of the part, and is transmitted to the rendering pipeline in the rendering module 102 for mapping for layered rendering. Optionally, the digital human application 101 and the rendering module 102 can be implemented by the same software module.

[0159] In an embodiment provided by the present invention, the layering method in the layered coding and layered rendering in the video processing method can be a secondary layering. For example, when the cloud application 101 is a cloud game, the first-level layer can be a background layer, a character layer, a special effect layer, a reflection layer, an icon layer, and a text layer, and the second-level layer can be a color layer, a texture layer, a highlight layer, a shadow layer, etc., that is, each first-level layer can be further layered at the second level. For example, the game developer can set the background layer and the character layer as the basic layer in the first-level layer, and the color layer as the basic layer in the second-level layer. Then the color layer in the background layer and the color layer in the character layer are both rendered with high priority and cannot be lowered by the user. As a result, the scalability of each layer in the video processing method proposed in the embodiment of the present invention is increased.

[0160] In other embodiments provided by the present invention, the video processing method and video processing system provided by the embodiments of the present invention can be applied to cloud games / remote games / online games, digital human services, cloud rendering / remote rendering / online rendering, cloud film and television production / remote film and television production / online film and television production and other fields. The present invention does not limit the application scenarios.

[0161] In summary, the embodiments of the present invention can utilize layered rendering content for layered encoding, avoid mutual influence of different layers of content during whole frame encoding, match rendering quality with encoding quality, reduce resource waste, and enhance picture stability. Compared with traditional video processing solutions, the coupling degree between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of other contents. Furthermore, the rendering effect set by the user can be fed back to the encoding link and penetrate into the rendering part. At the same time, the encoding layer can also be dynamically scaled according to the resource status of the current video processing system, saving resources and reducing information redundancy caused by layered rendering and layered encoding, which improves the quality of rendering and encoding while improving the user experience as a whole.

[0162] See also Fig.12 , Fig.12 is a schematic diagram of the structure of a video processing device provided by an embodiment of the present invention. The video processing device 700 can be implemented by software, hardware or a combination of both to become part or all of a video processing server. Fig.12 The video processing device 700 includes: a rendering module 701, an encoding module 702, a transmission module 703, an instruction receiving module 704, a resource status acquisition module 705, and an estimation module 706.

[0163] The rendering module 701 is used to render and generate original video data. The original video data is rendered by the original model data corresponding to the original video data. The original model data is generated by the cloud application based on the application operation instructions sent by the terminal device through the communication connection. The original video data includes M rendering layers, M≥2.

[0164] The encoding module 702 is used to encode M rendering layers in the original video data into i encoding layers to generate a target video bitstream, wherein a first part of the rendering layers in the M rendering layers is encoded as a first encoding layer, a second part of the rendering layers in the M rendering layers is encoded as a second encoding layer... an i-th part of the rendering layers in the M rendering layers is encoded as an i-th encoding layer, and M≥i≥1.

[0165] The transmission module 703 is used to send the target video code stream to the terminal device, and is also used to send resource occupancy information and / or picture effect information to the terminal device.

[0166] The instruction receiving module 704 is used to receive a picture effect configuration instruction, which is used to turn on / off the effect of a target rendering layer among the M rendering layers, and is also used to receive a picture effect configuration instruction, which is used to enhance / weaken the effect of the target rendering layer among the M rendering layers.

[0167] The resource status acquisition module 705 is used to obtain resource status information, where the resource status information is used to indicate the status of one or more of the rendering pipeline, encoder, communication connection and terminal device; and is also used to determine, based on the resource status information, that the resource status can satisfy the requirement of sending at least m original enhancement layers, where m≤i-1.

[0168] The estimation module 706 is used to estimate resource occupancy information and / or picture effect information based on the picture effect configuration instruction, wherein the resource occupancy information is used to describe the resource occupancy of the terminal device or the communication connection after the original video data is encoded according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after the original video data is encoded according to the picture effect configuration instruction.

[0169] Among them, the above-mentioned rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the rendering module is introduced below by taking the rendering module as an example. Similarly, the implementation of the video receiving module, the video decoding module, the information acquisition module, the strategy determination module, and the video sending module can refer to the implementation of the rendering module.

[0170] As an example of a software functional unit, a rendering module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing instance may be one or more. For example, the rendering module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0171] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0172] As an example of a hardware functional unit, the rendering module may include at least one computing device, such as a server, etc. Alternatively, module A may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0173] The multiple computing devices included in the rendering module can be distributed in the same region or in different regions. The multiple computing devices included in the rendering module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the rendering module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0174] It should be noted that, in other embodiments, the rendering module can be used to execute any step in the video processing method, and the encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module can also be used to execute any step in the video processing method. The steps that the rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module are responsible for implementing can be specified as needed. By respectively implementing different steps in the video processing method through the rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module, all functions of the video processing device can be realized.

[0175] In an embodiment of the present invention, during the video processing process, the layered rendering content can be used for layered encoding to avoid the mutual influence of different layers of content during the whole frame encoding, so that the rendering quality and the encoding quality match, reduce resource waste, and enhance the stability of the picture. Compared with the traditional video processing scheme, the coupling degree between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of the remaining content. At the same time, the coding layer can also be dynamically scaled according to the resource status of the current video processing system, saving resources, reducing the information redundancy caused by layered rendering and layered encoding, and improving the quality of rendering and encoding as a whole. Improve the user experience. It should be noted that: when the video processing device provided in the above embodiment processes video data, only the division of the above functional modules is used as an example. In actual applications, the above function allocation can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video processing device provided in the above embodiment and the video processing method embodiment belong to the same concept, and its specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0176] See also Fig.13 , Fig.13 1 is a schematic diagram of a computing device provided by an embodiment of the present invention, wherein the computing device 1000 includes: a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The embodiment of the present invention does not limit the number of processors and memories in the computing device 1000.

[0177] When implementing the technical solution provided by the embodiment of the present invention, the computing device 1000 may be a server that performs video processing tasks.

[0178] The bus 1002 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (eg, the memory 1006, the processor 1004, and the communication interface 1008).

[0179] The processor 1004 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0180] The memory 1006 may include a volatile memory, such as a random access memory (RAM). The processor 1004 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0181] In the embodiment of the present invention, the memory 1006 stores executable program codes, and the processor 1004 executes the executable program codes to respectively implement the functions of the aforementioned video receiving module 801, the video decoding module 802, the information acquisition module 803, the strategy determination module 804, the video processing module 805, and the video sending module 806, thereby implementing Figure 2 That is, the memory 1006 stores the video processing method for executing the above Figure 2 The video processing method shown in the figure is an instruction. It should be noted that Fig.13 Only the above-mentioned Figure 3 The program code of the video processing method shown is used as an example for explanation.

[0182] The communication interface 1008 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1000 and other devices or a communication network.

[0183] An embodiment of the present invention further provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0184] like Fig.14As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same Figure 3 Instructions for the video processing method shown.

[0185] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the video processing method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the video processing method. Figure 2 Instructions for the video processing method shown.

[0186] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the video processing device. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more of the aforementioned rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module.

[0187] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc. Fig.15 A possible implementation is shown, such as Fig.15 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device.

[0188] Using computing device clusters to achieve the above Fig.15 As an example, in the video processing method shown in Fig.15 In the implementation shown, the memory 1006 in the computing device 1000A stores instructions for executing the functions of the rendering module, the encoding module, and the transmission module. Meanwhile, the memory 1006 in the computing device 1000B stores instructions for executing the functions of the instruction receiving module, the resource status acquisition module, and the estimation module.

[0189] Fig.15 The connection method between the computing device clusters shown can be based on the consideration that the video processing method provided by the present invention may need to process a large amount of video data, so it is considered to implement multiple functions in different computing devices, and the functions implemented by the instruction receiving module 704, the estimation module 706, and the resource status acquisition module 705 are handed over to the computing device 1000B for execution, so as to reduce the computing amount of the computing device 1000A.

[0190] It should be understood that Fig.15The functions of the computing device 1000A shown in FIG. 1000A may also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B may also be completed by multiple computing devices 1000.

[0191] The embodiment of the present invention also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Fig.11 and Fig.15 Connection mode of computing device cluster. The difference is that the memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the video processing method.

[0192] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the video processing method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the video processing method.

[0193] The embodiment of the present invention also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above Figure 3 The video processing method shown.

[0194] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above Figure 3 The video processing method shown.

[0195] It should be understood that the "multiple" mentioned herein refers to two or more than two. In the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0196] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present invention are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method, characterized in that: Applied to a video processing device, the video processing device and the cloud application are arranged in at least one data center, a communication connection is established between the at least one data center and a terminal device, the method comprising: Rendering to generate original video data, wherein the original video data is rendered by original model data corresponding to the original video data, wherein the original model data is generated by the cloud application based on an application operation instruction sent by the terminal device through the communication connection, and the original video data includes M rendering layers, where M≥2; Encoding the M rendering layers in the original video data into i coding layers to generate a target video bitstream, wherein a first portion of the rendering layers in the M rendering layers is encoded as a first coding layer, a second portion of the rendering layers in the M rendering layers is encoded as a second coding layer, and ... an i-th portion of the rendering layers in the M rendering layers is encoded as an i-th coding layer, M≥i≥1; Send the target video code stream to the terminal device.

2. The method according to claim 1, characterized in that The first partial rendering layer is the first rendering layer among the M rendering layers, the second partial rendering layer is the second rendering layer among the M rendering layers, ... the i-th partial rendering layer is the i-th rendering layer among the M rendering layers, and the method further includes: Receive a picture effect configuration instruction, where the picture effect configuration instruction is used to turn on / off an effect of a target rendering layer among the M rendering layers; The rendering generates raw video data, including: According to the picture effect configuration instruction, start / stop rendering the target rendering layer, so as to start / stop encoding the encoding layer corresponding to the target rendering layer.

3. The method according to claim 1, characterized in that The method further comprises: Receive a picture effect configuration instruction, where the picture effect configuration instruction is used to enhance / weaken the effect of a target rendering layer among the M rendering layers; The rendering generates raw video data, including: According to the picture effect configuration instruction, the rendering quality of rendering the target rendering layer is increased / decreased.

4. The method according to any one of claims 1 to 3, characterized in that: The first coding layer among the i coding layers is an original base layer, and the other coding layers among the i coding layers are original enhancement layers, and the method further includes: Obtaining resource status information, where the resource status information is used to indicate a status of one or more of a rendering pipeline, an encoder, the communication connection, and the terminal device; Determine, according to the resource status information, that the resource status is sufficient to send at least m of the original enhancement layers, where m≤i-1; The original base layer and the m original enhancement layers are reconstructed into a target base layer, and the first bitstream includes a bitstream corresponding to the target base layer.

5. The method according to claim 4, characterized in that After reconstructing the original base layer and the m original enhancement layers into the target base layer, the method further includes: Determine that the resource status cannot satisfy the requirement of sending at least m of the original enhancement layers, but can satisfy the requirement of sending at least n of the original enhancement layers, n <m; Reconstructing the original base layer and n original enhancement layers into the target base layer; Stop rendering the n+2th rendering layer, the n+3th rendering layer, and so on the m+1th rendering layer.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: Based on the picture effect configuration instruction, estimate resource occupancy information and / or picture effect information, wherein the resource occupancy information is used to describe the resource occupancy of the terminal device or the communication connection after the original video data is encoded according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after the original video data is encoded according to the picture effect configuration instruction; The resource occupancy information and / or the picture effect information is sent to the terminal device.

7. A video processing method, characterized in that: Applied to a video processing device, the method comprises: Receive a target video bitstream, where the target video bitstream is generated by encoding original video data, where the original video data includes M rendering layers, the target video bitstream includes at least one coding layer, where the at least one coding layer includes a base layer and / or at least one enhancement layer, and where the i-th coding layer in the target video bitstream is obtained by encoding some rendering layers in the M rendering layers, where M≥2, M≥i≥1; The i-th coding layer is decoded into the partial rendering layer corresponding to the i-th coding layer, so as to decode the target video code stream.

8. The method according to claim 7, characterized in that The method further comprises: Send resource status information, wherein the resource status information is used to determine that the resource status can satisfy the sending of at least m original enhancement layers, m≤M-1, and the resource status information includes the status information of the terminal device, and the status information of the terminal device includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay.

9. A video processing device, characterized in that: The video processing device and the cloud application are arranged in at least one data center, a communication connection is established between the at least one data center and the terminal device, and the video processing device comprises: a rendering module, configured to render and generate original video data, wherein the original video data is rendered by original model data corresponding to the original video data, wherein the original model data is generated by the cloud application based on an application operation instruction sent by the terminal device through the communication connection, and the original video data includes M rendering layers, where M≥2; an encoding module, configured to encode the M rendering layers in the original video data into i encoding layers to generate a target video bitstream, wherein a first portion of the rendering layers in the M rendering layers is encoded as a first encoding layer, a second portion of the rendering layers in the M rendering layers is encoded as a second encoding layer, and ... an i-th portion of the rendering layers in the M rendering layers is encoded as an i-th encoding layer, M≥i≥1; A transmission module is used to send the target video code stream to the terminal device.

10. The device according to claim 9, characterized in that The first partial rendering layer is the first rendering layer among the M rendering layers, the second partial rendering layer is the second rendering layer among the M rendering layers, ... the i-th partial rendering layer is the i-th rendering layer among the M rendering layers, and the device further includes: An instruction receiving module, used for receiving a picture effect configuration instruction, wherein the picture effect configuration instruction is used for turning on / off the effect of a target rendering layer among the M rendering layers; The rendering module is further used for: According to the picture effect configuration instruction, start / stop rendering the target rendering layer, so as to start / stop encoding the encoding layer corresponding to the target rendering layer.

11. The device according to claim 9, characterized in that The device also includes: An instruction receiving module, used for receiving a picture effect configuration instruction, wherein the picture effect configuration instruction is used for enhancing / weakening the effect of a target rendering layer among the M rendering layers; The rendering module is further used for: According to the picture effect configuration instruction, the rendering quality of rendering the target rendering layer is increased / decreased.

12. The device according to any one of claims 9 to 11, characterized in that: The first coding layer among the i coding layers is an original base layer, and the other coding layers among the i coding layers are original enhancement layers, and the device further includes: a resource status acquisition module, configured to acquire resource status information, wherein the resource status information is used to indicate the status of one or more of the rendering pipeline, the encoder, the communication connection, and the terminal device; and further configured to determine, based on the resource status information, that the resource status is sufficient to send at least m of the original enhancement layers, where m≤i-1; The encoding module is further used for: The original base layer and the m original enhancement layers are reconstructed into a target base layer, and the first bitstream includes a bitstream corresponding to the target base layer.

13. The device according to claim 12, characterized in that The device also includes: The resource status acquisition module is further configured to determine that the resource status cannot satisfy the requirement of sending at least m original enhancement layers, but can satisfy the requirement of sending at least n original enhancement layers after the encoding module reconstructs the original base layer and the m original enhancement layers into the target base layer, wherein n <m; The encoding module is further used for: Reconstructing the original base layer and n original enhancement layers into the target base layer; The rendering module is further used for: Stop rendering the n+2th rendering layer, the n+3th rendering layer, and so on the m+1th rendering layer.

14. The device according to any one of claims 10 to 13, characterized in that: The device also includes: an estimation module, configured to estimate resource occupancy information and / or picture effect information based on the picture effect configuration instruction, wherein the resource occupancy information is used to describe the resource occupancy of the terminal device or the communication connection after the original video data is encoded according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after the original video data is encoded according to the picture effect configuration instruction; The transmission module is further used for: The resource occupancy information and / or the picture effect information is sent to the terminal device.

15. A video processing device, characterized in that: Applied to a video processing device, the device comprising: A receiving module, configured to receive a target video code stream, wherein the target video code stream is generated by encoding original video data, wherein the original video data includes M rendering layers, the target video code stream includes at least one coding layer, wherein the at least one coding layer includes a base layer and / or at least one enhancement layer, and the i-th coding layer in the target video code stream is obtained by encoding some rendering layers in the M rendering layers, M≥2, M≥i≥1; A decoding module is used to decode the i-th coding layer into the partial rendering layer corresponding to the i-th coding layer, so as to decode the target video code stream.

16. The device according to claim 15, characterized in that The device also includes: A sending module is used to send resource status information, wherein the resource status information is used to determine that the resource status can satisfy the sending of at least m original enhancement layers, m≤M-1, and the resource status information includes the status information of the terminal device, and the status information of the terminal device includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay.

17. A cloud application system, characterized in that: The cloud application system includes a cloud application, a rendering device and an encoding device, wherein the cloud application, the rendering device and the encoding device are arranged in at least one data center, wherein: The cloud application is used to generate the model data; The rendering device is used to render the model data to generate original video data, wherein the original video data includes M rendering layers, where M≥2; The encoding device is used to encode the M rendering layers in the original video data into i encoding layers, M≥i≥1, to generate a target video code stream.

18. A computing device cluster, comprising at least one computing device, characterized in that: Each computing device includes a processor and memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the video processing method according to any one of claims 1 to 6, or the video processing method according to any one of claims 7 to 8.

19. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the video processing method according to any one of claims 1 to 6, or the video processing method according to any one of claims 7 to 8.

20. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the video processing method according to any one of claims 1 to 6, or the video processing method according to any one of claims 7 to 8.

Images