Video processing method and related apparatus

Abstract

The present application discloses a video processing method and a related apparatus. The method is applied to a video processing apparatus, the apparatus and a cloud application are arranged in a data center, and a communication connection is established between the data center and a terminal device. The method comprises: performing rendering to generate original video data, wherein the original video data is obtained by rendering original model data corresponding to the original video data, the original model data is generated by the cloud application on the basis of an application operation instruction that is sent by the terminal device by means of the communication connection, and the original video data comprises M rendering layers; and coding the M rendering layers in the original video data into i coding layers, so as to generate a target video code stream, wherein a first part of rendering layers among the M rendering layers is coded as a first coded layer, ..., and an i-th part of rendering layers is coded as an i-th coded layer, M≥i≥1. According to the described method, hierarchical coding is performed on the basis of a rendering result having visual attributes and obtained by hierarchical rendering, reducing the interaction between layers, enhancing the controllability of the picture effect.

Description

Video processing method and related device Technical Field

[0001] The present application relates to the field of network technology, and in particular to a video processing method and related devices. Background Art

[0002] Layered coding, also known as scalable video coding (SVC), is used to encode video signals into layers and output multi-layer streams including a base layer and an enhancement layer. When bandwidth resources are insufficient, only the base layer stream is transmitted and decoded. When bandwidth resources are sufficient, the enhancement layer stream can be transmitted and decoded to improve the video decoding quality.

[0003] In related technologies, the layered coding encoder divides the video into time, space, and quality, encodes the original video data obtained from the rendering end to obtain the basic layer and the enhancement layer, and sends the video code stream including the basic layer and the enhancement layer to the terminal device according to the network's transmission capability. The terminal device is equipped with a decoding module to decode the video code stream, reconstruct the video frame and display it.

[0004] However, due to the loss caused by data compression during rendering encoding and transmission, the layers in this video processing method will affect each other, resulting in the rendered content being unable to be intuitively controlled by the user, affecting the user experience.

[0005] Summary of the Invention

[0006] This application provides a video processing method and related device that uses layered rendering information for layered encoding to reduce the mutual influence between image layers, allowing users to flexibly control the image rendering results and maximize the benefits of layered rendering. The technical solution is as follows:

[0007] In a first aspect, a video processing method is provided, which is applied to a video processing device, wherein the video processing device and a cloud application are arranged in at least one data center, and a communication connection is established between the at least one data center and a terminal device. The method includes: rendering to generate original video data, where the original video data is rendered by original model data corresponding to the original video data, and the original model data is generated by the cloud application based on an application operation instruction sent by the terminal device through the communication connection, and the original video data includes M rendering layers, where M ≥ 2; encoding the M rendering layers in the original video data into i coding layers to generate a target video stream, wherein a first part of the rendering layers in the M rendering layers is encoded as a first coding layer, a second part of the rendering layers in the M rendering layers is encoded as a second coding layer…an i-th part of the rendering layers in the M rendering layers is encoded as an i-th coding layer, where M ≥ i ≥ 1; and sending the target video stream to the terminal device.

[0008] It can be seen that in the process of rendering and encoding the original model data to generate a video code stream, the video processing device first renders the original model data into rendering data containing M rendering layers, and encodes some of the rendering layers into a coding layer, and encodes a total of i coding layers to complete layered coding. Among them, some rendering layers are one or more rendering layers in the M rendering layers, but not all of the M rendering layers. Optionally, a certain rendering layer in the M rendering layers can be rendered into a coding layer, or multiple rendering layers in the M rendering layers can be rendered into a coding layer. Thus, layered coding is performed based on the rendering layers obtained by layered rendering, which reduces the mutual influence between layers in the picture and avoids the loss of overall details of the picture due to encoding and transmission. In addition, the user's control over the picture can be directly reflected in the final picture through rendering, encoding, and transmission, which enhances the controllability of the picture effect and improves the user experience.

[0009] According to a possible implementation of the first aspect, wherein the first partial rendering layer is the first rendering layer among M rendering layers, the second partial rendering layer is the second rendering layer among the M rendering layers, and the i-th partial rendering layer is the i-th rendering layer among the M rendering layers, the method further includes: receiving a picture effect configuration instruction, the picture effect configuration instruction being used to turn on / off an effect of a target rendering layer among the M rendering layers; and rendering to generate raw video data, including: starting / stopping rendering the target rendering layer according to the picture effect configuration instruction, so as to start / stop encoding the coding layer corresponding to the target rendering layer.

[0010] Specifically, the video processing device encodes a rendering layer among the M rendering layers into a coding layer in a one-to-one correspondence, and the picture effect configuration instruction received by the video processing device directly indicates whether to turn on or off the effect of a rendering layer. Therefore, the adjustment instruction sent by the user for a certain picture layer can not only be directly executed in the rendering stage, but also be encoded into the coding layer corresponding to the picture layer or rendering layer in the coding stage, thereby avoiding the loss of overall picture details due to resource status fluctuations during the coding and transmission stages when adjusting the picture layer or rendering layer. For example, when a user turns on a certain picture layer, the picture layer is then rendered into the corresponding rendering layer, encoded into the corresponding coding layer and transmitted to the user, and will not affect the video display effect in terms of overall picture quality such as frame rate and resolution, thereby enhancing the user's controllability of the picture. Optionally, the user can perform picture effect configuration operations on the picture effect configuration interface displayed on the terminal device, and then send a picture effect configuration instruction to the video processing device. Optionally, the screen effect adjustment instruction sent by the user can be displayed on the screen effect configuration interface as a checkbox or drop-down box for turning a certain screen layer on or off, a checkbox for clicking and dragging the effect level of a certain screen layer, or a checkbox for adjusting the priority of different screen layers, etc. Optionally, the screen layer name displayed on the screen effect configuration interface is different from the corresponding rendering layer name.

[0011] According to a possible implementation of the first aspect, the method further includes: receiving a picture effect configuration instruction, where the picture effect configuration instruction is used to enhance / weaken the effect of a target rendering layer among the M rendering layers; and rendering to generate original video data, including: increasing / decreasing the rendering quality of the target rendering layer according to the picture effect configuration instruction.

[0012] Specifically, the picture effect configuration instruction received by the video processing device indicates whether to increase or decrease the rendering quality of a certain rendering layer. In this way, when the video processing device receives the picture effect configuration instruction, it can adjust the rendering intensity of a certain rendering layer in the rendering stage according to the instruction. Moreover, since the subsequent encoding stage will encode based on the rendering layer as the dimension, the impact of the picture effect configuration instruction on the rendering layer can be directly reflected in the final presented picture through rendering, encoding, and transmission, thereby enhancing the controllability of the picture effect and improving the user experience.

[0013] According to a possible implementation of the first aspect, among the i coding layers, the first coding layer is the original base layer, and the other coding layers among the i coding layers are the original enhancement layers. The method further includes: obtaining resource status information, where the resource status information is used to indicate the status of one or more of the rendering pipeline, encoder, communication connection, and terminal device; determining, according to the resource status information, that the resource status can satisfy sending at least m original enhancement layers, where m ≤ i - 1; reconstructing the original base layer and the m original enhancement layers into a target base layer, and the first bitstream includes the bitstream corresponding to the target base layer.

[0014] That is, based on hierarchical coding, by combining the status information of one or more of the resources such as the rendering pipeline, encoder, communication connection, and terminal device, the coding layers are dynamically scaled by merging enhancement layers or splitting the base layer. For example, when the resource status can satisfy sending at least m enhancement layers, the m enhancement layers and the base layer are merged. Optionally, it can be determined according to the resource status information that the long-term stable resource status within a period of time can satisfy sending a certain number of enhancement layers, and these enhancement layers and the base layer are selected to be merged. Thus, while ensuring the complete transmission of the information of these coding layers, the bitrate consumption caused by the poor decorrelation between layers in hierarchical rendering and hierarchical coding itself is saved.

[0015] According to a possible implementation of the first aspect, after reconstructing the original base layer and the m original enhancement layers into a target base layer, the method further includes: determining that the resource status cannot satisfy sending at least m original enhancement layers, but can satisfy sending at least n original enhancement layers, where n < m; reconstructing the original base layer and the n original enhancement layers into a target base layer; stopping rendering the (n + 2)-th rendering layer, the (n + 3)-th rendering layer... the (m + 1)-th rendering layer.

[0016] That is, based on hierarchical coding, by combining the state information of one or more of the resources such as the rendering pipeline, the encoder, the communication connection, and the terminal device, the coding layer is dynamically scaled by merging the enhancement layer or splitting the base layer. For example, after confirming that the resource state can satisfy sending multiple enhancement layers and merging these enhancement layers and the coding layer, that is, at this time, the base layer is already the base layer merged with the enhancement layer. If it is determined according to the resource state information at this time that the resource state can only satisfy sending fewer enhancement layers. For example, previously, m original enhancement layers (i.e., the initial, unmerged enhancement layers) could be sent, and now only n enhancement layers can be sent, where n < m. Then the system reconstructs the original base layer and n enhancement layers into the target base layer. In other words, the new base layer that was previously merged with m enhancement layers is split into the above target base layer and m - n layers. Optionally, it can be determined according to the resource state information that the long-term stable resource state within a period of time can satisfy sending a certain number of enhancement layers, and these enhancement layers and the base layer are selected to be merged. Thus, while ensuring the complete transmission of the information of these coding layers, the bitrate consumption caused by the poor decorrelation between layers in hierarchical rendering and hierarchical coding itself is saved.

[0017] According to a possible implementation manner of the first aspect, the method further includes: based on the picture effect configuration instruction, estimating the resource occupancy information and / or the picture effect information, where the resource occupancy information is used to describe the resource occupancy situation of the terminal device or the communication connection after encoding the original video data according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after encoding the original video data according to the picture effect configuration instruction; sending the resource occupancy information and / or the picture effect information to the terminal device.

[0018] That is, since the resource capabilities of the rendering pipeline, encoder, communication connection and terminal equipment are relatively fixed, when the video processing device receives the picture effect configuration instruction, it can estimate the resource usage of the video processing system in the current resource state when applying the above picture effect configuration instruction based on the configuration indicated by the instruction and various resource status information, such as CPU occupancy, GPU occupancy, memory occupancy, bandwidth occupancy, etc., and send this estimated information to the terminal device. Optionally, this estimated information is displayed on the resource configuration interface provided to the user. When the picture effect configuration instruction received by the video processing device indicates that when the picture effect configuration is greatly improved, the resource status is insufficient to support the effect configuration, and further, it will cause the layered coding to lose some picture details by discarding some enhancement layers in the short term. For example, when the user sets a lower priority for the water wave effect layer in the rendering layer, then when other high-priority picture effect configurations are improved, resulting in an increase in the bit rate and the bandwidth being unable to support the sending of the low-priority enhancement layer, the water wave effect layer will be discarded first during the layered encoding and transmission process. At this time, the picture effect information can be described as a weakened water wave effect or a water wave effect turned off. Optionally, this estimated information is displayed on the resource configuration interface provided to the user.

[0019] In a second aspect, a video processing method is provided, which is applied to a video processing device. The method includes: receiving a target video stream, where the target video stream is generated by encoding original video data, where the original video data includes M rendering layers, and the target video stream includes at least one coding layer, where the at least one coding layer includes a base layer and / or at least one enhancement layer, and where the i-th coding layer in the target video stream is obtained by encoding a portion of the M rendering layers, where M ≥ 2 and M ≥ i ≥ 1; and decoding the i-th coding layer into a portion of the rendering layer corresponding to the i-th coding layer, to decode the target video stream.

[0020] Specifically, the video processing device is used to decode the target code stream, which is generated through layered rendering and layered coding, and a certain coding layer of the layered coding of the target code stream is obtained by encoding a partial rendering layer in the rendering layer obtained by layered rendering, that is, the layered coding is encoded with the rendering layer obtained by layered rendering as the dimension. Furthermore, the video processing device is equipped with a decoder to decode the received coding layer into the corresponding partial rendering layer to decode the target video code stream. Optionally, the partial rendering layer may include one or more rendering layers of the M rendering layers, but not all rendering layers. Optionally, the video processing device is located in the user's terminal device.

[0021] According to a possible implementation of the second aspect, the method also includes: sending resource status information, the resource status information is used to determine whether the resource status can satisfy the sending of at least m original enhancement layers, m≤M-1, the resource status information includes status information of the terminal device, and the status information of the terminal device includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay.

[0022] Specifically, the video processing device collects and transmits resource status information from the terminal device, including one or more of CPU occupancy, layered decoding overhead, and layered decoding latency. Optionally, the video processing device transmits this resource status information to another video processing device performing encoding and rendering via a communication connection. This allows the other video processing device to confirm, based on the resource status information, that the resource status is sufficient to transmit at least m original enhancement layers.

[0023] In a third aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method in the first aspect. The video processing device includes at least one module, and the at least one module is used to implement the video processing method provided in the first aspect.

[0024] In a fourth aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method in the second aspect. The video processing device includes at least one module, and the at least one module is used to implement the video processing method provided in the second aspect.

[0025] In a fifth aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory;

[0026] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the video processing method described in the first aspect or the video processing method described in the second aspect.

[0027] In a sixth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the video processing method described in the first aspect or the video processing method described in the second aspect.

[0028] In a seventh aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the video processing method described in the first aspect above, or executes the video processing method described in the second aspect above.

[0029] The technical effects obtained in the above-mentioned third to seventh aspects are similar to the technical effects obtained by the corresponding technical means in the first and second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] FIG1 is a schematic diagram of the architecture of a video processing system provided in an embodiment of the present application;

[0032] FIG2 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0033] FIG3 is a flow chart of a video processing method provided in an embodiment of the present application;

[0034] FIG4 is a schematic diagram of the architecture of an encoding module provided in an embodiment of the present application;

[0035] Figures 5a-5e are schematic diagrams of a cloud application configuration interface provided in an embodiment of the present application;

[0036] Figures 6a-6c are schematic diagrams of a layered coding implementation method provided in an embodiment of the present application;

[0037] 7a-7c are schematic diagrams of the architecture of an encoding module provided in an embodiment of the present application;

[0038] FIG8 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0039] FIG9 is a flow chart of a video processing method provided in an embodiment of the present application;

[0040] FIG10 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0041] FIG11 is a schematic diagram of the architecture of another video processing system provided in an embodiment of the present application;

[0042] FIG12 is a schematic structural diagram of a video processing device provided in an embodiment of the present application;

[0043] FIG13 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0044] FIG14 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0045] FIG15 is a schematic diagram of a network connection relationship of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] First, some terms and related technologies involved in the present invention are explained with reference to the accompanying drawings to facilitate understanding by those skilled in the art.

[0049] Layer rendering: Split the overall rendering process into several different layers, render each layer separately and save the corresponding rendering results. Layered rendering saves different screen elements or data in different layers, and then stitches them together to form the final rendering effect. For example, the rendering can be divided into foreground layer, mid-ground layer, and background layer according to the content; for example, it can be divided into color layer, highlight layer, shadow layer, reflection layer, refraction layer, luminous layer, etc. according to the visual properties of the rendering; for example, it can be layered according to visual saliency. Reasonable layered rendering not only improves speed, but also makes it convenient to adjust various effects in the later stage.

[0050] Scalable video coding (SVC): Also known as scalable video coding or hierarchical video coding, it is scalable and layerable, capable of being divided into different levels of frame rate, resolution, and quality. A single encoding process can output multiple streams, divided into a base layer and an enhancement layer, suitable for different terminals and network conditions. The base layer occupies relatively little bandwidth resources and ensures basic video quality. The addition of enhancement layers to the base layer can achieve better frame rate, resolution, or quality. SVC coding scalability is divided into temporal scalability, spatial scalability, and quality scalability. Temporal scalability refers to the decomposition of a video stream into information representing different frame rates, spatial scalability refers to the decomposition of a video stream into information representing different resolutions, and quality scalability refers to the decomposition of pixel values ​​into different levels.

[0051] Base layer: The lowest layer of data in layered coding. It contains basic video information, such as resolution, frame rate, and color. This allows the decoder to properly decode the main content of the video, but the video quality may be lower. Base layer data can be decoded independently or combined with data from other enhancement layers to improve video quality, resolution, or frame rate. Base layer data typically uses a lower bit rate, making it suitable for transmission over bandwidth-limited networks or playback on lower-performance devices.

[0052] Enhancement layer: Data other than the lowest layer in layered coding. It contains additional information about the video, such as a higher frame rate, resolution, quality, etc., and can be combined with the data in the base layer to improve the performance and experience of the video. The data in the enhancement layer cannot be decoded independently and needs to rely on the data in the base layer or other enhancement layers. The data in the enhancement layer can be dynamically adjusted according to different network conditions, terminal capabilities, or user needs to achieve scalability and adaptability of the video. There can be multiple enhancement layer data, forming different levels, and each level can correspond to a dimension in the time domain, spatial domain, or quality.

[0053] The rendering pipeline, a core component of real-time graphics, is the process of converting three-dimensional (3D) scene data into two-dimensional (2D) images. It typically consists of multiple stages, each of which processes the input data and passes the output to the next stage. The rendering pipeline generally includes the application stage, geometry stage, rasterization stage, and fragment stage.

[0054] Before explaining the video encoding method provided by the embodiment of the present invention in detail, the application scenario of the embodiment of the present invention is first introduced.

[0055] Layered rendering technology is widely used in the production of games, film and television special effects, and animation. There are many forms of layered rendering. For example, it can be divided into foreground layer, midground layer, and background layer according to the content of the rendering; for example, it can be divided into color layer, highlight layer, shadow layer, reflection layer, refraction layer, luminous layer, etc. according to the visual properties of the rendering. Layered coding refers to the division in time, space or quality, and the output of multi-layer code streams including the base layer and the enhancement layer. The data of the base layer can enable the decoder to decode the basic video content completely normally, thereby ensuring that the decoding end receives smooth video images that can be viewed normally when the channel is limited or the channel environment is complex. When the channel environment is good or the channel resources are abundant, the enhancement layer data can be transmitted to improve the video quality.

[0056] Video encoding compresses raw video, reducing its data size and bitrate without negatively impacting its visual quality. Encoding services are built on cloud computing or the vendor's physical servers. Cloud-based encoding services can save users the high investment of purchasing, setting up, and managing encoding hardware and software, and avoid complexities such as configuration optimization and encoding parameter adaptation. Furthermore, the elastic scalability of cloud services can meet actual encoding business needs.

[0057] In cloud gaming and digital human applications, the main process from receiving commands to layered rendering, layered encoding, and transmission is as follows: the user sends signaling data from the terminal, such as game character control instructions and digital human movement instructions. The application generates model data based on the signaling data and sends it to the rendering device for rendering. This model data is then rendered to obtain raw video data, i.e., raw video frames. The encoder then encodes the raw video data to obtain an encoded video stream. This video stream is then transmitted to the user terminal device via a streaming protocol, so that the decoding module installed on the user terminal device decodes the received video stream and displays it on the user terminal device. In this process, rendering can be achieved through layered rendering, where the rendering device renders the raw model data separately according to color layers, highlight layers, shadow layers, reflection layers, refraction layers, and luminescence layers. Encoding can also be achieved through layered encoding, where the encoder performs temporal, spatial, or quality-layered encoding on the raw video data to obtain a base layer and at least one enhancement layer. These layers are selectively transmitted to the user side based on network conditions. Finally, the terminal device 20 performs layered decoding to obtain and display the video frames.

[0058] However, when related technologies perform layered rendering and layered encoding on videos, layered encoding and network transmission affect the effect of layered rendering. Layered rendering technology enhances the possibility of post-control and adjustment of different levels of effects, but after layered encoding encodes a basic layer and at least one enhancement layer, due to factors such as network status, not all enhancement layers will be transmitted during transmission, which may result in loss of rendering content, and thus waste computing resources consumed in the rendering process. When the picture of a certain layer fluctuates, it will affect the effects of other layers. In addition, when layered rendering is performed locally, users can intuitively adjust the picture according to visual attributes. For example, in a local application that provides fine picture control, the user inputs a command to turn on the highlight effect, and the highlight effect can be immediately presented in the picture without being affected by the encoding method or network status. However, when the rendering side is separated from the user side, such as when layered rendering occurs in the cloud or on the application manufacturer's server, the user-side experience cannot be consistent with the rendering side. That is, when the user wants to adjust the effect of a certain visual layer based on intuitive visual attributes, the overall visual effect of the picture may be affected due to encoding or network fluctuations, making it difficult for the user to make subjective judgments and resulting in a poor experience.

[0059] Based on this, an embodiment of the present invention provides a video processing method, which performs layered encoding by utilizing layered rendering content, thereby avoiding the mutual influence of different layers of content during whole-frame encoding, matching the rendering quality with the encoding quality, reducing resource waste, and enhancing the stability of the picture. Compared with traditional video processing solutions, the degree of coupling between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of other contents. At the same time, the encoding layer can also be dynamically scaled according to the resource status of the current video processing system, saving resources and reducing the information redundancy caused by layered rendering and layered encoding. Overall, it improves the quality of rendering and encoding while enhancing the user experience.

[0060] It should be understood that the video processing scenarios in the above examples are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Ordinary technicians in this field can know that with the emergence of new video processing scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0061] Please refer to Figure 1, which is a schematic diagram of a video processing system architecture provided by an embodiment of the present invention. As shown in Figure 1, the video processing system includes a service-side device 10, a terminal device 20 and a communication connection 30, wherein the service-side device 10 includes a 3D application 101, a layered rendering module 102, an encoding module 103, and a control module 104, and the terminal device 20 includes a system and hardware 201, a control module 202, and a decoding module 203. The service-side device 10 and the terminal device 20 can communicate through the communication connection 30, thereby enabling users to use cloud games, digital humans and other applications, control them in real time, and display real-time images on the terminal device 20.

[0062] In one embodiment provided by the present invention, a user 40 uses a 3D application 101 via an application installed on a terminal device 30, such as a personal computer or smartphone, or uses the 3D application 101 via a webpage. The application 101 may be a digital human application or a cloud gaming application, and the present invention does not limit this. The 3D application 101 may be replaced with a 2D application, and the present invention does not limit this.

[0063] Next, the video processing method is explained by taking the cloud gaming application service scenario as an example. The service-side device 10 serves as the cloud gaming screen video transmitter, receives the user's 40 operation instructions, such as game operation instructions for manipulating the game character to perform different actions or move on the scene, to generate original model data, and transmits it to the rendering module 102 for layered rendering. Multiple rendering layers are generated during the layered rendering process. Afterwards, the various rendering layers rendered by the rendering module 102 are integrated into the original video data, which includes the frame-level images generated in real time by the rendering module 102, that is, the original video frames. The encoding module encodes the video frames generated by the rendering module into a base layer and an enhancement layer, frame by frame. For example, when using a spatially scalable layered coding scheme, the encoding module 103 encodes the multiple original video frames to produce multiple images of different resolutions. The base layer code stream encodes a low-resolution image. Adding the enhancement layer code stream to this low-resolution image allows the decoding module 203 in the terminal device 20 to decode the high-resolution image. Thus, the original video data is encoded into a binary video code stream by the video encoder on the service side. This video code stream is transmitted to the terminal device 20 via the communication connection 30. The terminal device 20 is equipped with a decoding module 203, which decodes the video code stream to obtain reconstructed video data, which includes multiple reconstructed video frames. Due to layered coding, when network bandwidth fluctuates, different numbers and levels of coding layers are sent based on the needs and capabilities of the receiver, so that the receiver can decode video signals of different quality or resolution based on the received coding layers.

[0064] Since the original model data and rendering data will inevitably be compressed during the rendering, encoding, and transmission processes, and since the content is not perceived during encoding, the loss during data compression is completely determined by the encoder. In this process, the quality of different contents of the picture will affect each other due to direct content overlap and indirect bit rate competition. For example, if the text on the picture overlaps with the background, even if the text itself does not change between multiple frames, the text will become blurred if the complexity of the overlapping background texture increases or the motion intensity increases, which reduces the user experience.

[0065] In response to the above problems, an embodiment of the present invention proposes a video processing method, which uses layered rendering content for layered encoding to avoid the mutual influence of different layers of content during whole-frame encoding, so that the rendering quality and encoding quality match, reduce resource waste, and enhance the stability of the picture. Compared with traditional video processing solutions, the coupling degree between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of other content. The encoding link becomes a function that can be controlled by application users and application developers. The rendering effect set by the application user can not only be fed back to the encoding link, but also penetrate into the rendering part, so that the rendering and encoding as well as the computing power of the network can be matched.

[0066] At the same time, when a video processing device based on layered rendering and layered coding processes video data, the following problems may arise: both layered rendering and layered coding have some drawbacks in decorrelation. Layered coding requires higher coding complexity and bitrate because it requires different coding processes for each layer of information, such as quantization, transformation, and entropy coding, and also requires ensuring synchronization and compatibility between the layers. At the same time, layered coding also introduces additional redundant information, such as correlation and dependency between layers, as well as redundant coding within layers, which increases the bitrate after encoding. Layered rendered images themselves have strong correlations, such as the association between color and texture, or the influence between depth and illumination. Therefore, either layered coding or layered rendering itself will waste a certain amount of bitrate.

[0067] To address the above issues, an embodiment of the present invention proposes a video processing method. When the encoding module performs layered encoding on the original video data, it can maximize the bit rate savings when the resource status is stable by dynamically merging and decomposing the encoding layers.

[0068] Please refer to Figure 2, which is a schematic diagram of a video processing system architecture provided by an embodiment of the present invention. As shown in Figure 2, the video processing system includes a service-side device 10, a terminal device 20 and a communication connection 30, wherein the service-side device 10 includes a 3D application 101, a layered rendering module 102, an encoding module 103, and a control module 104, and the terminal device 20 includes a system and hardware 201, a control module 202, and a decoding module 203. The service-side device 10 and the terminal device 20 can communicate through the communication connection 30, thereby enabling users to use cloud games, digital humans and other applications, control them in real time, and display real-time images on the terminal device 20.

[0069] In one embodiment provided by the present invention, a user 40 uses a 3D application 101 via an application installed on a terminal device 30, such as a personal computer or smartphone, or uses the 3D application 101 via a webpage. The application 101 may be a digital human application or a cloud gaming application, and the present invention does not limit this. The 3D application 101 may be replaced with a 2D application, and the present invention does not limit this.

[0070] Next, taking the cloud game application service scenario as an example, the video processing method is explained. The service side device 10 serves as the cloud game screen video transmitter, receives the operation instructions of the user 40, such as the game operation instructions for controlling the game character to make different actions or move on the scene, to generate the original model data, and transmits it to the rendering module 102 for layered rendering. In this process, the service side device 10 configuration determines the rendering order and rendering quality of each rendering layer. The configuration can be preset by the system, such as the system presets the original template and makes the initial configuration according to the user's hardware and network status; it can also be manually configured by the user, such as the user sets to turn on or off a certain screen effect, adjust the quality level of a certain screen effect, or set the priority of different screen effects, or the user selects a set of rendering configurations in the rendering template library provided by the application; or the service side can adaptively adjust the rendering configuration in real time according to information such as the network status and hardware status, and the present invention does not limit this. Then, the rendering module 102 can adjust the specific rendering effect of a certain rendering layer according to the configuration sent by the user 40 or the system preset configuration. For example, in a scene manually configured by the user, when the user 40 inputs a picture control instruction to enhance the highlight effect, the rendering module 102 correspondingly increases the rendering intensity of the highlight layer.

[0071] Thus, the rendering module 102 renders the original video data to obtain multiple rendering layers, obtains the original video data, and sends it to the encoding module 103. The encoding module 103 encodes the rendering layers in the original video data into encoding layers one by one, that is, for example, the color layer is encoded as a basic layer, the highlight layer is encoded as an enhancement layer, the shadow layer is encoded as another enhancement layer, and the reflection layer is encoded as another enhancement layer... At this time, the enhancement layer encoded with the highlight layer and the like is superimposed on the basic layer encoded by the color layer, so that the decoding module 203 in the terminal device 20 can decode and obtain a high-resolution image. Thus, the original video data is layered encoded by the video encoder on the service side using the information of layered rendering to generate a binary video code stream, which is transmitted to the terminal device 20 via the communication connection 30. The terminal device 20 is equipped with a decoding module 203 to decode the video code stream to obtain reconstructed video data, which includes multiple reconstructed video frames. The encoding module of this embodiment uses the information of layered rendering to perform layered encoding, so that when layered encoding sends encoding layers of different numbers and levels, the mutual influence between the layers is reduced, and the overall picture quality details of the video will not be lost due to changes in a certain layer. At the same time, the rendering effect set by the user can be fed back to the encoding link and penetrate into the rendering part, and can control the rendering layer corresponding to the encoding layer according to the subjectively perceptible picture configuration, meeting the user's needs for controlling picture effects and quality in scenarios such as cloud gaming.

[0072] Based on the video processing system architecture of FIG2 , the video processing method provided by an embodiment of the present invention is introduced in detail below by taking the cloud gaming application service scenario as an example in combination with FIG3 .

[0073] Please refer to FIG3 below, which is a flow chart of a video processing method provided by an embodiment of the present invention. As shown in FIG3 , the method includes but is not limited to the following steps:

[0074] S101 . The user 40 inputs an application operation instruction through the terminal device 20 .

[0075] User 40 enters application operation instructions through terminal device 20. For example, user 40 is a user of 3D application 101, i.e., a cloud game player. Terminal device 20 is a device such as a personal computer or smartphone. 3D application 101 can be an application installed on terminal device 20, or its page can be accessed through a browser on terminal device 20, although this is not limited in the present invention. The application operation instructions can be used to control the movement of a game character, control the release of a game character's skills, or trigger game cutscenes, etc. The application operation instructions can be triggered by touching a touch screen, clicking a mouse, or inputting a keyboard, etc., although this is not limited in the present invention.

[0076] S102. The terminal device 20 sends an application operation instruction.

[0077] Specifically, the terminal device 20 encapsulates the above-mentioned application operation instructions into a data message and sends it to the service-side device 10 through the communication connection 30. Specifically, the communication connection 30 can be the Internet, and the service-side device can be provided by a physical server or a data center. Furthermore, the above-mentioned physical server or data center can be one or more; it can also be provided by cloud computing resources such as virtual machines and containers; in other embodiments provided by the present invention, the service-side device 10 can refer to a collection of functional units implemented by software functions, for example, including codes running on multiple hosts / virtual machines / containers, and the present invention is not limited to this.

[0078] S103. The 3D application 101 generates original model data.

[0079] For example, the 3D application 101 is a cloud gaming application that generates raw model data based on the game operation instructions sent by the user 40. Optionally, the raw model data may include information such as geometry, topology, color, texture, and animation. Optionally, in one embodiment provided by the present invention, the raw model data may be scene graphics data or animation data, wherein the scene graphics data includes information such as objects, lighting, materials, and textures in the game or animation, and is used to generate visual effects in three-dimensional space. This data can be converted from vertex coordinates to pixels on the screen through a graphics rendering pipeline, and then the pixels are shaded by a shader and finally output as a video. The animation data includes motion information of characters, objects, special effects, etc. in the game or animation, and is used to generate dynamic effects. This data can be bound together by skeletal animation, and then the pose of each frame is calculated by interpolation, and finally output as a video. The raw model data can be stored in different formats, such as OBJ, FBX, GLTF, etc., and the present invention is not limited to this. In other embodiments provided by the present invention, 3D applications can be replaced with 2D applications, that is, the original model data can be two-dimensional or three-dimensional, and the present invention does not limit this. Specifically, when the instruction received by the cloud game is a game character control instruction, the original model data may include model data for the game character to perform corresponding actions and model data for background animation; when the instruction received by the cloud game is a cutscene trigger instruction, the original model data may include a fixed-length cutscene model data, such as, including generating appearance model data based on the appearance of the game user-defined character, and combining skeletal drive data to generate the original model data of the character, etc. The present invention does not limit the generation trigger conditions, generation method and content of the original model data.

[0080] S104. The 3D application 101 sends the original model data.

[0081] For example, the 3D application 101 sends the original model data to the rendering module 102 for further rendering to generate a video image that can be presented.

[0082] S105. The rendering module 102 renders the original model data into multiple rendering layers.

[0083] First, the rendering module 102 can layer the objects in the scene according to certain standards. Optionally, it can be divided into a foreground layer, a midground layer, and a background layer according to the rendered content. Specifically, if the 3D application 101 is a digital human application, it can be divided into a skin layer, an eye layer, a mouth shape layer, an expression layer, a hair layer, and an accessory layer according to the rendered content. It can also be combined with the embodiment of Figure 2, which is a schematic diagram of a layered rendering method provided by an embodiment of the present invention. As shown in the figure, the rendering module 102 divides the rendered content into a color layer, a highlight layer, a shadow layer, a reflection layer, a refraction layer, a luminous layer, etc. according to visual attributes. Then, independent rendering settings are performed on each rendering layer, including a renderer, rendering parameters, rendering channels, etc. Finally, each rendering layer is rendered to obtain a rendered image of each rendering layer.

[0084] The rendering module 102 can also be called a rendering pipeline. Optionally, the rendering module 102 may include one or more of a ray tracing module, a rasterization module, and a shader module; it may also include one or more of a vertex processing module, a texture mapping module, an ambient occlusion rendering module, a high dynamic range rendering module, and a post-processing module; specifically, the rendering module may also include one or more of rendering modules with specific functions such as a particle rendering module, a lighting rendering module, a shadow rendering module, and a water wave rendering module, and the present invention is not limited to this.

[0085] S106. The encoding module 103 encodes the multiple rendering layers into multiple encoding layers to generate a video stream.

[0086] The encoding module 103 encodes the multiple rendering layers generated by the rendering module 102 in step S105 into multiple coding layers. Specifically, the encoding module 103 encodes one or more rendering layers into one coding layer to generate a video stream containing multiple coding layers. For example, when the rendering layers are divided into a color layer, a highlight layer, a shadow layer, a reflection layer, a refraction layer, and a luminous layer, the encoding module 103 may encode the color layer as a base layer, the highlight layer as a first enhancement layer, and the shadow layer as a second enhancement layer. Alternatively, the encoding module 103 may encode the color layer and the shadow layer as a base layer, the highlight layer as a first enhancement layer, and the reflection layer and the refraction layer as a second enhancement layer. For another example, when the game is a soldier battle game and the rendered screen is soldiers fighting on the grass, the rendering layer can be divided into a character layer (including soldiers), a background layer (including grass), and a text layer (including battle broadcasts). The encoding module 103 can encode the character layer as the first basic layer and the background layer as the second basic layer, so that the user end can present the characters and background to ensure a relatively complete game screen, and so on.

[0087] An embodiment of an encoding module 103 for encoding multiple rendering layers into multiple coding layers can be combined with FIG4 , which is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. As shown in FIG4 , the result of rendering the highlight layer is fed to the encoder and encoded as a base layer, and the result of rendering the shadow layer to the reflection layer is fed to the encoder and encoded as an enhancement layer. Taking the shadow layer as an example, the encoder subtracts the reconstructed value of the previous layer encoder from the rendered video frame of the current layer to obtain a residual, which is then encoded as an enhancement layer. This process is repeated for subsequent layers to obtain multiple enhancement layers.

[0088] S107. The encoding module 103 sends the video code stream.

[0089] Specifically, the video code stream includes multiple coding layers generated by the coding module 103 in step S106.

[0090] S108 . The user 40 inputs a picture effect configuration instruction through the terminal device 20 to turn on / off the effect of the target rendering layer.

[0091] Specifically, the user 40 inputs a screen effect configuration instruction on the interface provided by the cloud application 101 via the client installed on the terminal device 20 or a browser capable of accessing the cloud application via the communication connection 30 to turn on or off the effect of the target rendering layer. An embodiment of the user inputting the configuration instruction can be combined with Figures 5a to 5d, which are schematic diagrams of a screen effect configuration interface provided by an embodiment of the present invention. As shown in Figures 5a to 5d, in Figure 5a, the items that can be configured by the user on the screen effect configuration interface include turning on or off a certain layer of screen and the effect level of a certain layer of screen, wherein the effect level can be divided into very low, low, medium, high, very high, etc., and the present invention does not limit this.

[0092] In other embodiments provided by the present invention, as shown in FIG5b , the items that can be configured by the user in the screen effect configuration interface are screen layer priority preferences, such as highlight priority, shadow priority, etc. In FIG5c , the screen layers displayed in the screen effect configuration interface can be divided according to the content of the screen, such as player characters, non-player characters, water surface effects, particle effects, ground and light, etc. In other embodiments provided by the present invention, as shown in FIG5d , the cloud application 101 can be a digital human application, and the screen layers displayed in the screen effect configuration interface can be divided according to the content of the screen into skin, eyes, mouth shape, facial expressions, hair and accessories, etc. Therefore, the user can turn on or off the corresponding rendering layer by clicking on or off, adjusting the level, setting the priority, etc. to configure the desired screen effect, and the present invention does not limit this.

[0093] Optionally, the video processing method and video processing device provided by the embodiments of the present invention provide users with a picture effect configuration interface and a picture effect configuration interface in the form of an end-side application or a browser, etc. The picture effect configuration interface is used to receive a picture effect configuration instruction sent by the user, so that the service side modifies the rendering configuration and encoding configuration based on the picture configuration instruction. Optionally, when the occupancy rate of the hardware resources on the user side, such as the CPU, graphics card, etc., is close to saturation, or the bandwidth bit rate difference of the network is less than the threshold preset by the system or set by the user, the user attempts to send a picture effect configuration instruction to enhance the highlight layer rendering quality, but the system resources are insufficient to support the enhanced quality, the system estimates the resource occupancy or actual picture effect after enhancing the highlight layer rendering quality based on the instruction, and displays it in real time on the user's configuration interface for the user to determine whether to apply the configuration.

[0094] In another embodiment provided by the present invention, the developer of application 101 can use the video processing system of Figure 2 and the video processing method of Figure 3 provided in the embodiment of the present invention to assist in application design when designing the application. For example, an illustration of an embodiment of the present invention can be referred to Figure 5e, which is a schematic diagram of a design interface that can be provided to application developers. For example, when the application 101 is a cloud game, the developer of the cloud game can set different rendering coding priorities and rendering coding configurations for different layers of content during design to ensure the quality of high-priority content; at the same time, the developer can adjust the complexity and priority of the rendering through real-time debugging. For example, if the background, character, special effects, and text are layered, if the developer finds that the rendering quality of the background grass is low, the video picture presented by the video processing method in the embodiment of the present invention has the characteristics of end-to-end intuitive feedback, that is, it indicates that the texture complexity of the current background layer itself is high, or the rendering configuration or priority corresponding to the background layer is low, the developer can specifically reduce the texture complexity of the picture itself, or adjust the rendering configuration of the background layer, avoiding the situation in the related technology where all layers of the picture affect each other during the encoding and transmission stages, making it impossible to intuitively locate the problem; the control module 104 can count data of different layers and different contents (such as bit rate, encoding complexity, encoding quality), which can be used as a reference for developers in cloud game design to help developers better analyze problems. Optionally, the developer may provide an initial configuration or an optimal configuration. For example, the developer may set the color layer as the base layer and the texture layer as the first enhancement layer, and this configuration cannot be manually changed by the game player. In another example, the developer may provide an initial priority for each rendering layer for the game player's reference. For example, a recommended configuration prompt may be given when the player first enters the game, and the player may be guided to apply the configuration with one click. The embodiment of the present invention does not limit the method of providing the recommended configuration.

[0095] S109. The terminal device 20 sends a screen effect configuration instruction.

[0096] After receiving the configuration instruction from the user in step S108, the terminal device 20 sends the screen effect configuration instruction to the server-side device. The screen effect configuration instruction is used to instruct the rendering module to render according to the configuration. Specifically, it can be sent to the rendering module 102. In other embodiments provided by the present invention, the terminal device 20 can first send the configuration information generated according to the effect configuration instruction in step S108 to the control module 104 of the server-side device 10, and then the control module 104 generates the screen effect configuration instruction and sends it to the rendering module 102. The present invention is not limited to this.

[0097] Optionally, in other embodiments provided by the present invention, the screen effect configuration instruction can be generated by the control module based on resource status information. For example, the service side device 10 can generate a recommended initial screen effect configuration for the user 40 based on resource status information such as the processor configuration of the terminal device 20, the bandwidth size and stability of the communication connection 30, or the rendering and encoding capabilities of the service side device 10, or generate a recommended real-time screen effect configuration during the application operation, and send it to the rendering module 102. The present invention does not limit this.

[0098] S110 . The rendering module 102 starts / stops rendering the target rendering layer according to the picture effect configuration instruction.

[0099] Specifically, after receiving the screen effect configuration instruction, the rendering module 102 starts rendering according to the screen effect configuration. For example, if the screen effect configuration instruction indicates to start rendering of the highlight layer, the rendering module starts to use the renderer, rendering parameters and rendering channels related to the highlight layer to render and generate the highlight layer of the picture.

[0100] S111 . The rendering module 102 starts / stops sending the target rendering layer.

[0101] For example, if the rendering module 102 receives a picture effect configuration instruction to render and generate a highlight layer, the highlight layer is sent to the encoding module 103 for the next step of encoding; if the rendering module 102 receives a picture effect configuration instruction to stop generating the highlight layer, the highlight layer is stopped from being sent to the encoding module 103.

[0102] S112. The encoding module 103 starts / stops encoding the encoding layer corresponding to the target rendering layer.

[0103] Specifically, if the rendering module 102 starts rendering the highlight layer and sends the highlight layer to the encoding module 103, the encoding module 103 starts encoding the coding layer corresponding to the highlight layer. For example, if the encoding module encodes a rendering layer into a corresponding coding layer, the highlight layer is encoded into a base layer or an enhancement layer; if the encoding module encodes multiple rendering layers into corresponding coding layers, the highlight layer and the other rendering layers are encoded into a base layer or an enhancement layer.

[0104] S113. Collect resource status information.

[0105] The control module 104 collects resource status information. Specifically, the sources from which the control module 104 can collect the resource status information may include: the encoder in the encoding module 103 in the service-side device 10, the rendering pipeline in the rendering module 102, the transmission network corresponding to the communication connection 30, and the terminal device 20, etc. An architecture for the control module 104 to collect resource status information from these modules can refer to the embodiment provided in Figure 8. For example, the resource status information may include information such as the encoding bit rate, encoding complexity, encoding content scene, motion intensity, and layered encoding overhead in the encoder; it may also include information such as the rendering content complexity, rendering content changes, and 3D application rendering methods in the rendering pipeline; it may also include information such as the real-time network bandwidth, network fluctuations, packet loss rate, and bandwidth bit rate difference in the transmission network; it may also include information such as the CPU occupancy rate, layered decoding overhead, and layered decoding delay in the terminal device. That is, the resource status information is information that reflects the operating status of the video processing system including the user-side and service-side devices. It is worth noting that in other embodiments provided by the present invention, the reference resource status information for the control module to configure the layered rendering and layered coding strategies includes one or more of the information in the above examples, and other indicators reflecting the software and hardware status and network status of the rendering and coding system can also be used as a reference. The present invention does not limit this.

[0106] S113a. Send terminal device resource status information.

[0107] Specifically, the terminal device 20 sends the terminal device resource status information to the control module 104 in the server-side device 10. For example, the resource status information may include CPU occupancy, layered decoding overhead, layered decoding delay, and the like.

[0108] S113b. Send rendering pipeline resource status information.

[0109] Specifically, the rendering module 102 sends rendering pipeline resource status information to the control module 104 in the server-side device 10. For example, the status information may include rendering content complexity, rendering content changes, 3D application rendering mode, and the like.

[0110] S113c. Send transmission network resource status information.

[0111] Specifically, the communication connection 30 sends the transmission network resource status information to the control module 104 in the service-side device 10. For example, the information may include real-time network bandwidth, network fluctuation, packet loss rate, bandwidth-to-bitrate difference, and the like.

[0112] S113d. Send encoder resource status information.

[0113] Specifically, the encoding module 103 sends encoder resource status information to the control module 104 in the server-side device 10. For example, the information may include encoding bit rate, encoding complexity, encoding content scene, motion intensity, layered encoding overhead, and the like.

[0114] S114. According to the resource status information, determine that the resource status can satisfy the sending of at least m enhancement layers

[0115] Specifically, the control module 104 determines, based on the collected resource status information, that the current resource status of the video processing system is sufficient to send at least m enhancement layers.

[0116] For example, the bitrate or bandwidth-to-bitrate difference can be used to determine whether, when bitrate fluctuations are small, at least m enhancement layers can be sent for the next step of merging coding layers. Large bitrate fluctuations indicate that the system cannot stably send m enhancement layers, and thus cannot merge them. Similarly, the bandwidth-to-bitrate difference is used for judgment. The bandwidth-to-bitrate difference is the difference between the bandwidth and the bitrate. If the bandwidth consistently exceeds the bitrate over a period of time, at least m enhancement layers can be sent for the next step of merging coding layers.

[0117] For another example, the judgment can be made based on the encoding complexity. When the encoding complexity is high, it is judged that more resources will be consumed, and the number of enhancement layers that can be sent in the next step will be reduced. For another example, the judgment can be made based on the content scenario. When the currently running application 101 is a racing game, because the racing game screen is rich in details and changes rapidly, the bit rate is high, and accordingly, the number of enhancement layers that can be sent is relatively small; when the currently running application 101 is document editing software, because document editing software has more pure colors and less changes, the bit rate is low, and accordingly, the number of enhancement layers that can be sent is relatively large; for another example, the judgment can be made based on the rendering content complexity. For example, when the user-side configuration requires its frame rate to be stable at 90fps and the rendering capability of the server-side device is constant, if the rendering complexity is high, the resource status may not be sufficient to support the rendering of m+2 enhancement layers, but can only support the rendering of m enhancement layers. In this way, the control module 104 determines that the resource status of the current video processing system can meet the requirements of sending at least m enhancement layers.

[0118] In summary, the control module 104 determines, based on the collected resource status information, that the current resource capabilities of each module in the system are sufficient to support the transmission of at least m enhancement layers. Optionally, the control module 104 determines, by statistically analyzing changes in the resource status information over a certain period of time, that the current resource capabilities of each module in the system are sufficient to support the transmission of at least m enhancement layers over a long period of time.

[0119] S115. Send a coding layer merge instruction.

[0120] Specifically, the control module 104 sends a coding layer merging instruction to the encoding module 103 to instruct the encoding module 103 to merge the original base layer and the m enhancement layers. In other embodiments provided by the present invention, when the control module 104 determines in step S114 based on the resource status information that the current resource status cannot satisfy the requirement of sending at least m enhancement layers but can only satisfy the requirement of sending at least n enhancement layers, the control module 104 may send a coding layer decomposition instruction in step S115 to instruct the encoding module to decompose the merged base layer into a new base layer and mn enhancement layers.

[0121] S116. Reconstruct the original base layer and the m enhancement layers into a target base layer.

[0122] Specifically, after the encoding module 103 receives the coding layer merging instruction sent by the control module 104, the original base layer and m enhancement layers are reconstructed into the target base layer, and used as the base layer for layered coding. The remaining coding layers continue to be used as enhancement layers. The merged base layer reduces information redundancy compared to the multiple coding layers before the merger, so that in the long-term stable state, the encoder only sends one necessary base layer and a small number of enhancement layers, reducing the information entropy loss caused by layered rendering and layered coding, and saving hardware and network resources.

[0123] An embodiment for reconstructing the original base layer and multiple enhancement layers into a new base layer can be combined with Figures 6a to 6c, which are schematic diagrams of a method for reconstructing the original base layer and multiple enhancement layers provided by an embodiment of the present invention. As shown in the figure, the layered coding merging steps are as follows: First, as shown in Figure 6a, layer m, layer m+1, and layer m+2 are selected as the base layer and multiple enhancement layers to be merged; then, as shown in Figure 6b, the reconstruction value of the base layer is refreshed using the reconstruction value of the last enhancement layer, layer m+2; finally, as shown in Figure 6c, the base layer is moved to the position of the last enhancement layer, layer m+2, and the rendering result corresponding to this layer is encoded as the base layer, and then a new video frame is encoded based on the new reconstruction value. Optionally, the decoding process is similar to the encoding process, except that the merging and decomposition are only performed at the encoder level, and the rendering part is not involved. The decomposition process directly moves the base layer forward gradually.

[0124] An embodiment of a combined encoding module 103 for implementing layered coding can be combined with Figure 7a, which is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. Specifically, as shown in Figure 7a, in this embodiment, for example, the base layer is the color layer, and the enhancement layers are the texture layer, highlight layer, and shadow layer. The rendered color layer can be encoded using a universal encoder in the related art. The residual between the rendered texture layer and the color layer is subtracted. The color layer is added to this residual to encode the first enhancement layer in the coding layer. The encoder for this first enhancement layer is a proprietary encoder. That is, implementation is achieved using a non-compliant video encoder and a proprietary encoder, wherein the non-compliant video encoder requires updating the reconstruction value, while the proprietary encoder compresses the residual. Optionally, the residual can be compressed using a lossless method or using image compression methods in the related art.

[0125] When the control module 104 determines based on the resource status information that the color layer, texture layer and highlight layer are to be merged in the layered coding, the base layer reconstruction value needs to be updated with the enhancement layer corresponding to the highlight layer during the coding process as the base layer reference value in the coding process, so as to serve as the reference value when encoding the next frame.

[0126] Another embodiment of the encoding module 103 for implementing layered coding can be described in conjunction with FIG7b , which is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. Specifically, as shown in FIG7b , in this embodiment, when merging the color layer and texture layer, the merged frame directly references the base layer of the previous frame as the base layer reference value. That is, the coding layer corresponding to the highlight layer of the previous frame is directly used as the base layer reference value for inter-frame prediction. This encoding method is relatively simple to implement and can be implemented using both standard-compliant video encoders and proprietary encoders.

[0127] Another embodiment of the encoding module 103 for implementing layered coding can be combined with FIG7c, which is a schematic diagram of an encoder in a video processing device provided by an embodiment of the present invention. As shown in FIG7c, in this embodiment, the rendered color layer, texture layer, highlight layer, and shadow layer are all encoded using a video encoder. That is, each rendering layer is encoded as a video frame, with the color layer as the first frame, the texture layer as the second frame, and the highlight layer as the third frame. This can be implemented based on a fully compliant video encoder and private outer layer scheduling. The enhancement layer encoder is consistent with the previous layer encoder. When the control module 104 determines to merge multiple coding layers, the enhancement layer directly uses the previous layer to encode the video frame of the next layer, and uses the encoded enhancement layer code stream to refresh the base layer reference value. The encoder can set the motion vector to 0 and use fixed block partitioning to accelerate the encoding process.

[0128] S117. Send video stream.

[0129] Specifically, the encoding module 103 sends the encoded video stream, which includes the combined coding layers, to the user's terminal device 20 via the communication connection 30. Optionally, after the video stream is decoded by the decoder of the terminal device 20, the video image can be displayed on the display device included in the terminal device 20.

[0130] Please refer to Figure 9, which is a flow chart of a video processing method provided by an embodiment of the present invention. As shown in Figure 9, in an embodiment provided by the present invention, the control module may include a short-term layered coding control module 1041 and a long-term layered coding control module 1042, and the coding module 103 may include a short-term layered coding module 1031 and a long-term layered coding module 1032.

[0131] The short-term layered coding module 1031 controls frame-level coded data transmission and resource status information statistics. First, it receives the multi-layered codestream input from the coding module 103 and sends the base layer to the transport layer. Then, based on the real-time or statistical resource status information described above, it determines whether to send the next enhancement layer. This process repeats until no enhancement layer can be sent. Optionally, it also collects real-time information such as the number of enhancement layers sent, the bandwidth-to-bitrate difference, and the encoder's coding complexity. Based on this information, it calculates a fluctuation coefficient and sends it to the long-term layered coding module 1032 for reference.

[0132] The long-term layered coding module 1032 is used to control the generation and allocation of enhancement layers and base layers. When the long-term bandwidth can stably support the base layer and one or more enhancement layers, the module controlling the layers in the long-term layered coding module 1032 merges these base layers and enhancement layers into a new base layer, thereby increasing coding efficiency and ensuring coding stability. When the long-term bandwidth cannot stably support or the fluctuation coefficient is large, the module controlling the layers disassembles the base layer into new base layers and enhancement layers until the base layer can meet the requirements. At the same time, optionally, for enhancement layers and rendering layers that will not be used, the long-term layered coding module 1032 can stop the rendering pipeline in advance and control the number of enhancement layers, thereby saving rendering computing power.

[0133] In one embodiment provided by the present invention, the long-term layered coding module 1032 can accept a predefined template configured by the developer of the application 101. For example, the predefined template can configure information such as the layered rendering method of the application content, and based on this information, the entire system can be controlled to generate the most suitable initial rendering and coding configuration. At the same time, optionally, the long-term layered coding module 1032 can calculate the long-term stability information of the system over a period of time, such as the bit rate, computing power and other resource overheads required for different enhancement layers, and adjust it through the adaptive control module to achieve the best overall effect. In addition, this information can also be provided to the user side for reference, so that the user can participate in the adjustment of the system to achieve a better subjective experience.

[0134] FIG9 shows a flow chart of a method in which the control module 104 instructs the encoding module 103 to merge and decompose encoding layers. The video processing method includes the following steps:

[0135] S201. Send the base layer.

[0136] Specifically, the short-term layered coding module 1031 in the coding module 103 sends the base layer to the terminal device 20.

[0137] S202. Confirm that the resource status satisfies the transmission of the first enhancement layer, the second enhancement layer, and the third enhancement layer.

[0138] Specifically, the short-term layered coding control module 1041 in the control module 104 confirms that three enhancement layers can be sent currently through real-time resource information such as network bandwidth, and informs the short-term layered coding module 1031 .

[0139] S203. Send the first enhancement layer, the second enhancement layer, and the third enhancement layer.

[0140] Specifically, the short-term layered coding control module 1041 in the control module 104 instructs the short-term layered coding module 1031 to send the first enhancement layer, the second enhancement layer, and the third enhancement layer.

[0141] S204: Confirm that the long-term resource status can satisfy the transmission of the first enhancement layer and the second enhancement layer.

[0142] Specifically, the long-term layered coding control module 1042 in the control module 104 collects statistics on the long-term stable resource status of each system module to confirm that the long-term resource status is sufficient for transmitting the first and second enhancement layers, and notifies the long-term layered coding module 1032 of the information. For example, the resource status may fluctuate at this time, and the lowest value of the fluctuation may be sufficient for transmitting the first and second enhancement layers, but may not be sufficient for long-term stable transmission of the third enhancement layer. Optionally, the short-term layered coding control module makes real-time adjustments based on the network bandwidth status to determine whether the third enhancement layer can be transmitted.

[0143] S205. Merge the base layer, the first enhancement layer, and the second enhancement layer into a new base layer a.

[0144] Specifically, the long-term layered coding control module 1042 instructs the long-term layered coding module 1032 to merge the base layer and the first enhancement layer and the second enhancement layer into a new base layer a. An embodiment of the merging can refer to Figures 6a, 6b, 6c and Figures 7a, 7b, and 7c.

[0145] S206. Send the base layer a and the third enhancement layer.

[0146] Specifically, the long-term layered coding module 1032 sends the merged base layer a and the third enhancement layer to the terminal device 20.

[0147] S207. Confirm that the long-term resource status cannot satisfy the transmission of the second enhancement layer.

[0148] Specifically, the long-term layered coding control module 1032 determines that the long-term resource status cannot meet the requirements for transmitting the second enhancement layer, and notifies the long-term layered coding control module 1032. In other words, the long-term layered coding control module 1032 determines that the long-term resource status of the system cannot meet the requirements for transmitting the second enhancement layer based on the stable resource status information of the system over a period of time.

[0149] S208. Decompose the base layer a into the base layer b and the second enhancement layer.

[0150] Specifically, the long-term layered coding control module 1032 decomposes the base layer a into the base layer b and the second enhancement layer. Optionally, the original base layer and enhancement layer may be reconstructed into the base layer b and the second enhancement layer.

[0151] S209: Confirm that the resource status cannot meet the requirements for sending the third enhancement layer.

[0152] Specifically, the short-term layered coding control module 1041 in the control module 104 confirms that the resource status cannot meet the transmission of the third enhancement layer, and informs the short-term layered coding module 1031 in the coding module 103 .

[0153] S210. Send the base layer b and the second enhancement layer.

[0154] Specifically, the short-term layered coding module 1031 in the coding module 103 sends the base layer b and the second enhancement layer to the terminal device 20 .

[0155] Please refer to Figure 10, which is an architecture diagram of a video processing system provided by an embodiment of the present invention. As shown in Figure 10, in an embodiment provided by the present invention, the video processing system is implemented in an edge-cloud collaborative manner. The video processing system includes a service-side device 10a and a service-side device 10b, that is, the service-side device 10 in the video processing system shown in Figure 2 includes a service-side device 10a and a service-side device 10b, wherein the service-side device 10a is provided by one or more cloud data centers, and the service-side device 10b is provided by one or more edge cloud devices. The edge cloud is a technology that deploys cloud computing capabilities closer to the data source or user. The difference between it and the public cloud mainly lies in the deployment location, service objects, resource management and application scenarios. It can provide services nearby at the edge node, reduce network delay and jitter, and improve the response speed and quality of the service. It can also filter, compress, aggregate and other processes on the data at the edge node to reduce the amount of data transmission, save network bandwidth and traffic, and reduce network costs.

[0156] In a scenario provided by an embodiment of the present invention, application 101 is a role-playing game, and the rendering layer of the game includes a background layer, a character layer, a special effects layer, a reflection layer, an icon layer and a text layer. Specifically, the background layer can be the background in which the characters are fighting, such as grass, a beach, etc., the special effects layer can be the special effects when the characters use skills in battle, the reflection layer can be the reflection effect of objects such as characters and weapons, which is used to make the objects look more textured and realistic, and the icon layer and the text layer can be used to display game prompts, character names, player dialogues, etc.

[0157] In this embodiment, the original model data for rendering generated by the application can be distributed to the service-side device 10a and the service-side device 10b for rendering. For example, the background layer, character layer, special effects layer and reflection layer can be rendered, encoded and transmitted by the service-side device 10a, and the icon layer and text layer can be rendered, encoded and transmitted by the service-side device 10b. Since the content in the icon layer and the text layer is mostly high-frequency information, the complexity of its rendering is lower than the complexity of encoding and transmission, that is, the required computing power is lower, but the required bandwidth is higher. Therefore, this part of information is rendered and encoded on the edge device through edge-cloud collaboration. With the help of the better network quality and lower transmission bandwidth of the edge device, this information can be encoded with higher quality, which greatly improves the image quality of some content on the terminal side and solves the pain point problem of poor quality of information such as text.

[0158] Please refer to Figure 11, which is an architectural diagram of a video processing system provided by an embodiment of the present invention. As shown in Figure 11, in one embodiment provided by the present invention, cloud application 101 is specifically a digital human application 101. This digital human application 101 can use artificial intelligence (AI) reasoning technology to generate a virtual image that simulates human appearance, expression, voice, and language, and can perform multimodal interactions and services in different scenarios. In this embodiment, digital human application 101 uses voice-driven, key point generation, and other technologies to generate raw data such as mouth shape, face, and eye expression. This raw data can include dynamic effects and position information of the parts, and is transmitted to the rendering pipeline in rendering module 102 for mapping for layered rendering. Optionally, digital human application 101 and rendering module 102 can be implemented by the same software module.

[0159] In one embodiment provided by the present invention, the layering method in the layered encoding and layered rendering in the video processing method can be a secondary layering. For example, when the cloud application 101 is a cloud game, the first-level layer can be a background layer, a character layer, a special effect layer, a reflection layer, an icon layer, and a text layer, and the second-level layer can be a color layer, a texture layer, a highlight layer, a shadow layer, etc., that is, each first-level layer can be further layered at the second level. For example, the game developer can set the background layer and the character layer as the basic layer in the first-level layer, and the color layer as the basic layer in the second-level layer. Then the color layer in the background layer and the color layer in the character layer are both rendered with high priority and cannot be lowered in priority by the user. This increases the scalability of each layer in the video processing method proposed in the embodiment of the present invention.

[0160] In other embodiments provided by the present invention, the video processing method and video processing system provided by the embodiments of the present invention can be applied to cloud games / remote games / online games, digital human services, cloud rendering / remote rendering / online rendering, cloud film and television production / remote film and television production / online film and television production and other fields. The present invention does not limit the application scenarios.

[0161] In summary, the embodiments of the present invention can utilize layered rendering content for layered encoding, avoid the mutual influence of different layers of content during whole-frame encoding, match the rendering quality with the encoding quality, reduce resource waste, and enhance the stability of the picture. Compared with traditional video processing solutions, the degree of coupling between different contents is reduced, and when the complexity of a certain content increases, it will not cause drastic fluctuations in the quality of other contents. Furthermore, the rendering effect set by the user can be fed back to the encoding link and penetrate into the rendering part. At the same time, the encoding layer can also be dynamically scaled according to the resource status of the current video processing system, saving resources and reducing the information redundancy caused by layered rendering and layered encoding. Overall, the quality of rendering and encoding is improved while enhancing the user experience.

[0162] Please refer to Figure 12, which is a schematic diagram of the structure of a video processing device provided in an embodiment of the present invention. This video processing device 700 can be implemented as part or all of a video processing server using software, hardware, or a combination of both. Referring to Figure 12, this video processing device 700 includes a rendering module 701, an encoding module 702, a transmission module 703, a command receiving module 704, a resource status acquisition module 705, and an estimation module 706.

[0163] The rendering module 701 is used to render and generate original video data. The original video data is rendered by the original model data corresponding to the original video data. The original model data is generated by the cloud application based on the application operation instructions sent by the terminal device through the communication connection. The original video data includes M rendering layers, M≥2.

[0164] The encoding module 702 is configured to encode M rendering layers in the original video data into i coding layers to generate a target video bitstream, wherein a first portion of the M rendering layers is encoded as a first coding layer, a second portion of the M rendering layers is encoded as a second coding layer, and so on, an i-th portion of the M rendering layers is encoded as an i-th coding layer, where M≥i≥1.

[0165] The transmission module 703 is used to send the target video stream to the terminal device, and is also used to send resource occupancy information and / or picture effect information to the terminal device.

[0166] The instruction receiving module 704 is used to receive a picture effect configuration instruction, which is used to turn on / off the effect of a target rendering layer among the M rendering layers, and is also used to receive a picture effect configuration instruction, which is used to enhance / weaken the effect of the target rendering layer among the M rendering layers.

[0167] The resource status acquisition module 705 is used to obtain resource status information, where the resource status information is used to indicate the status of one or more of the rendering pipeline, encoder, communication connection, and terminal device; and is also used to determine, based on the resource status information, whether the resource status can satisfy the requirement of sending at least m original enhancement layers, where m≤i-1.

[0168] The estimation module 706 is used to estimate resource occupancy information and / or picture effect information based on the picture effect configuration instruction, wherein the resource occupancy information is used to describe the resource occupancy of the terminal device or the communication connection after the original video data is encoded according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after the original video data is encoded according to the picture effect configuration instruction.

[0169] The rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module can all be implemented via software or hardware. For example, the implementation of the rendering module will be described below using the rendering module as an example. Similarly, the implementation of the video receiving module, video decoding module, information acquisition module, policy determination module, and video sending module can refer to the implementation of the rendering module.

[0170] As an example of a software functional unit, a rendering module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the rendering module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0171] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0172] As an example of a hardware functional unit, a rendering module may include at least one computing device, such as a server. Alternatively, module A may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0173] The multiple computing devices included in the rendering module can be distributed in the same region or in different regions. The multiple computing devices included in the rendering module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the rendering module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0174] It should be noted that, in other embodiments, the rendering module can be used to execute any step in the video processing method, and the encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module can also be used to execute any step in the video processing method. The steps that the rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module are responsible for implementing can be specified as needed. By respectively implementing different steps in the video processing method through the rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module, all functions of the video processing device can be realized.

[0175] In embodiments of the present invention, during video processing, layered encoding can be performed using layered rendering content, avoiding the mutual influence of different layers of content during full-frame encoding. This ensures that rendering quality and encoding quality match, reduces resource waste, and enhances image stability. Compared to traditional video processing solutions, the coupling between different content is reduced, and when the complexity of one content increases, the quality of other content does not fluctuate drastically. Furthermore, the encoding layers can be dynamically scaled based on the resource status of the current video processing system, saving resources and reducing information redundancy caused by layered rendering and layered encoding. Overall, this improves rendering and encoding quality while enhancing the user experience. It should be noted that the video processing device provided in the above embodiments only uses the division of the above functional modules as an example to illustrate the processing of video data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to perform all or part of the functions described above. In addition, the video processing device provided in the above embodiments and the video processing method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0176] Please refer to Figure 13, which is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. The computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The embodiment of the present invention does not limit the number of processors and memories in the computing device 1000.

[0177] When implementing the technical solution provided by the embodiment of the present invention, the computing device 1000 may be a server that performs video processing tasks.

[0178] Bus 1002 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG13 shows only one line, but this does not imply a single bus or type of bus. Bus 1002 may include a path for transmitting information between various components of computing device 1000 (e.g., memory 1006, processor 1004, and communication interface 1008).

[0179] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0180] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0181] In this embodiment of the present invention, the memory 1006 stores executable program code, and the processor 1004 executes the executable program code to respectively implement the functions of the aforementioned video receiving module 801, video decoding module 802, information acquisition module 803, strategy determination module 804, video processing module 805, and video sending module 806, thereby implementing the video processing method shown in Figure 2. In other words, the memory 1006 stores instructions for executing the video processing method shown in Figure 2. It should be noted that Figure 13 only uses the program code of the video processing method shown in Figure 3 stored in the memory 1006 as an example for explanation.

[0182] The communication interface 1008 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1000 and other devices or a communication network.

[0183] An embodiment of the present invention further provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0184] As shown in Figure 14, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the video processing method shown in Figure 3.

[0185] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store some instructions for executing the video processing method. In other words, the combination of one or more computing devices 1000 can jointly execute instructions for executing the video processing method shown in Figure 2.

[0186] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, each used to execute a portion of the functions of the video processing apparatus. In other words, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more of the aforementioned rendering module, encoding module, transmission module, instruction receiving module, resource status acquisition module, and estimation module.

[0187] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN). FIG. 16 illustrates one possible implementation. As shown in FIG. 16 , two computing devices 1000A and 1000B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0188] Taking the video processing method shown in FIG. 16 as an example, the memory 1006 of computing device 1000A stores instructions for executing the functions of the rendering module, encoding module, and transmission module. Meanwhile, the memory 1006 of computing device 1000B stores instructions for executing the functions of the instruction receiving module, resource status acquisition module, and estimation module.

[0189] The connection method between the computing device clusters shown in Figure 15 can be based on the consideration that the video processing method provided by the present invention may need to process a large amount of video data, so it is considered to implement multiple functions in different computing devices, and the functions implemented by the instruction receiving module 704, the estimation module 706, and the resource status acquisition module 705 are handed over to the computing device 1000B for execution, so as to reduce the computing amount of the computing device 1000A.

[0190] It should be understood that the functions of the computing device 1000A shown in FIG15 may also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B may also be completed by multiple computing devices 1000.

[0191] Embodiments of the present invention also provide another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection relationship between computing device clusters in Figures 11 and 15. However, the memory 1006 in one or more computing devices 1000 in this computing device cluster can store the same instructions for executing the video processing method.

[0192] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store some instructions for executing the video processing method. In other words, the combination of one or more computing devices 1000 can jointly execute the instructions for executing the video processing method.

[0193] An embodiment of the present invention further provides a computer program product including instructions. This computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute the video processing method shown in FIG. 3 .

[0194] Embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the video processing method shown in FIG. 3 .

[0195] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0196] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present invention are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A video processing method, characterized in that: Applied to a video processing device, the video processing device and the cloud application are arranged in at least one data center, a communication connection is established between the at least one data center and a terminal device, the method comprising: Rendering to generate original video data, wherein the original video data is rendered by original model data corresponding to the original video data, wherein the original model data is generated by the cloud application based on an application operation instruction sent by the terminal device through the communication connection, and the original video data includes M rendering layers, where M≥2; Encoding the M rendering layers in the original video data into i coding layers to generate a target video bitstream, wherein a first portion of the rendering layers in the M rendering layers is encoded as a first coding layer, a second portion of the rendering layers in the M rendering layers is encoded as a second coding layer, and ... an i-th portion of the rendering layers in the M rendering layers is encoded as an i-th coding layer, M≥i≥1; Send the target video code stream to the terminal device.

2. The method according to claim 1, characterized in that The first partial rendering layer is the first rendering layer among the M rendering layers, the second partial rendering layer is the second rendering layer among the M rendering layers, ... the i-th partial rendering layer is the i-th rendering layer among the M rendering layers, and the method further includes: Receive a picture effect configuration instruction, where the picture effect configuration instruction is used to turn on / off an effect of a target rendering layer among the M rendering layers; The rendering generates raw video data, including: According to the picture effect configuration instruction, start / stop rendering the target rendering layer, so as to start / stop encoding the encoding layer corresponding to the target rendering layer.

3. The method according to claim 1, characterized in that The method further comprises: Receive a picture effect configuration instruction, where the picture effect configuration instruction is used to enhance / weaken the effect of a target rendering layer among the M rendering layers; The rendering generates raw video data, including: According to the picture effect configuration instruction, the rendering quality of rendering the target rendering layer is increased / decreased.

4. The method according to any one of claims 1 to 3, characterized in that: The first coding layer among the i coding layers is an original base layer, and the other coding layers among the i coding layers are original enhancement layers, and the method further includes: Obtaining resource status information, where the resource status information is used to indicate a status of one or more of a rendering pipeline, an encoder, the communication connection, and the terminal device; Determining, according to the resource status information, that the resource status is sufficient to send at least m of the original enhancement layers, where m≤i-1; The original base layer and the m original enhancement layers are reconstructed into a target base layer, and the first bitstream includes a bitstream corresponding to the target base layer.

5. The method according to claim 4, characterized in that After reconstructing the original base layer and the m original enhancement layers into the target base layer, the method further includes: Determine that the resource status cannot satisfy the requirement of sending at least m of the original enhancement layers, but can satisfy the requirement of sending at least n of the original enhancement layers, n <m; Reconstructing the original base layer and n original enhancement layers into the target base layer; Stop rendering the n+2th rendering layer, the n+3th rendering layer, and so on the m+1th rendering layer.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: Based on the picture effect configuration instruction, estimate resource occupancy information and / or picture effect information, wherein the resource occupancy information is used to describe the resource occupancy of the terminal device or the communication connection after the original video data is encoded according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after the original video data is encoded according to the picture effect configuration instruction; The resource occupancy information and / or the picture effect information is sent to the terminal device.

7. A video processing method, characterized in that: Applied to a video processing device, the method comprises: Receive a target video bitstream, where the target video bitstream is generated by encoding original video data, where the original video data includes M rendering layers, the target video bitstream includes at least one coding layer, where the at least one coding layer includes a base layer and / or at least one enhancement layer, and where the i-th coding layer in the target video bitstream is obtained by encoding some rendering layers in the M rendering layers, where M≥2, M≥i≥1; The i-th coding layer is decoded into the partial rendering layer corresponding to the i-th coding layer, so as to decode the target video code stream.

8. The method according to claim 7, characterized in that The method further comprises: Send resource status information, wherein the resource status information is used to determine that the resource status can satisfy the sending of at least m original enhancement layers, m≤M-1, and the resource status information includes the status information of the terminal device, and the status information of the terminal device includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay.

9. A video processing device, characterized in that: The video processing device and the cloud application are arranged in at least one data center, a communication connection is established between the at least one data center and the terminal device, and the video processing device comprises: a rendering module, configured to render and generate original video data, wherein the original video data is rendered by original model data corresponding to the original video data, wherein the original model data is generated by the cloud application based on an application operation instruction sent by the terminal device through the communication connection, and the original video data includes M rendering layers, where M≥2; an encoding module, configured to encode the M rendering layers in the original video data into i encoding layers to generate a target video bitstream, wherein a first portion of the rendering layers in the M rendering layers is encoded as a first encoding layer, a second portion of the rendering layers in the M rendering layers is encoded as a second encoding layer, and ... an i-th portion of the rendering layers in the M rendering layers is encoded as an i-th encoding layer, M≥i≥1; A transmission module is used to send the target video code stream to the terminal device.

10. The device according to claim 9, characterized in that The first partial rendering layer is the first rendering layer among the M rendering layers, the second partial rendering layer is the second rendering layer among the M rendering layers, ... the i-th partial rendering layer is the i-th rendering layer among the M rendering layers, and the device further includes: An instruction receiving module, used for receiving a picture effect configuration instruction, wherein the picture effect configuration instruction is used for turning on / off the effect of a target rendering layer among the M rendering layers; The rendering module is further used for: According to the picture effect configuration instruction, start / stop rendering the target rendering layer, so as to start / stop encoding the encoding layer corresponding to the target rendering layer.

11. The device according to claim 9, characterized in that The device also includes: An instruction receiving module, used for receiving a picture effect configuration instruction, wherein the picture effect configuration instruction is used for enhancing / weakening the effect of a target rendering layer among the M rendering layers; The rendering module is further used for: According to the picture effect configuration instruction, the rendering quality of rendering the target rendering layer is increased / decreased.

12. The device according to any one of claims 9 to 11, characterized in that: The first coding layer among the i coding layers is an original base layer, and the other coding layers among the i coding layers are original enhancement layers, and the device further includes: a resource status acquisition module, configured to acquire resource status information, wherein the resource status information is used to indicate the status of one or more of the rendering pipeline, the encoder, the communication connection, and the terminal device; and further configured to determine, based on the resource status information, that the resource status is sufficient to send at least m of the original enhancement layers, where m≤i-1; The encoding module is further used for: The original base layer and the m original enhancement layers are reconstructed into a target base layer, and the first bitstream includes a bitstream corresponding to the target base layer.

13. The device according to claim 12, characterized in that The device also includes: The resource status acquisition module is further configured to determine that the resource status cannot satisfy the requirement of sending at least m original enhancement layers, but can satisfy the requirement of sending at least n original enhancement layers after the encoding module reconstructs the original base layer and the m original enhancement layers into the target base layer, wherein n <m; The encoding module is further used for: Reconstructing the original base layer and n original enhancement layers into the target base layer; The rendering module is further used for: Stop rendering the n+2th rendering layer, the n+3th rendering layer, and so on the m+1th rendering layer.

14. The device according to any one of claims 10 to 13, characterized in that: The device also includes: an estimation module, configured to estimate resource occupancy information and / or picture effect information based on the picture effect configuration instruction, wherein the resource occupancy information is used to describe the resource occupancy of the terminal device or the communication connection after the original video data is encoded according to the picture effect configuration instruction, and the picture effect information is used to describe the picture effect displayed on the terminal device after the original video data is encoded according to the picture effect configuration instruction; The transmission module is further used for: The resource occupancy information and / or the picture effect information is sent to the terminal device.

15. A video processing device, characterized in that: Applied to a video processing device, the device comprising: A receiving module, configured to receive a target video code stream, wherein the target video code stream is generated by encoding original video data, wherein the original video data includes M rendering layers, the target video code stream includes at least one coding layer, wherein the at least one coding layer includes a base layer and / or at least one enhancement layer, and the i-th coding layer in the target video code stream is obtained by encoding some rendering layers in the M rendering layers, M≥2, M≥i≥1; A decoding module is used to decode the i-th coding layer into the partial rendering layer corresponding to the i-th coding layer, so as to decode the target video code stream.

16. The device according to claim 15, characterized in that The device also includes: A sending module is used to send resource status information, wherein the resource status information is used to determine that the resource status can satisfy the sending of at least m original enhancement layers, m≤M-1, and the resource status information includes the status information of the terminal device, and the status information of the terminal device includes one or more of CPU occupancy, layered decoding overhead, and layered decoding delay.

17. A cloud application system, characterized in that: The cloud application system includes a cloud application, a rendering device and an encoding device, wherein the cloud application, the rendering device and the encoding device are arranged in at least one data center, wherein: The cloud application is used to generate the model data; The rendering device is used to render the model data to generate original video data, wherein the original video data includes M rendering layers, where M≥2; The encoding device is used to encode the M rendering layers in the original video data into i encoding layers, M≥i≥1, to generate a target video code stream.

18. A computing device cluster, comprising at least one computing device, characterized in that: Each computing device includes a processor and memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the video processing method according to any one of claims 1 to 6, or the video processing method according to any one of claims 7 to 8.

19. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the video processing method according to any one of claims 1 to 6, or the video processing method according to any one of claims 7 to 8.

20. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the video processing method according to any one of claims 1 to 6, or the video processing method according to any one of claims 7 to 8.

Images