Video processing method, system and related device

Abstract

The invention provides a video processing method and system and a related device, and the method comprises the steps: obtaining a network state parameter at the moment when a video is rendered, determining a rendering parameter according to the network state parameter, and then carrying out the rendering of the video according to the rendering parameter, thereby obtaining a cloud rendering content. And finally, the cloud rendering content can be displayed in the terminal equipment of the user. According to the method and the device, the quality of the rendering content downloaded and extracted by the user can be improved, the consistency of the rendering content downloaded and extracted by the user and the cloud rendering content at the cloud end is ensured, and the rendering experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of media technology, and in particular to a video processing method, system and related devices. Background Art

[0002] Rendering refers to the process of using software to generate images from models, where the model is a description of a three-dimensional object using a strictly defined language or data structure, which includes geometry, viewpoint, texture, and lighting information. Images are digital images or bitmap images. Video processing in the cloud is a process of rendering videos using cloud computing technology. Users submit local video content to a remote server, which performs computing operations through remote computer cluster resources, renders and encodes the uploaded video content, and then transmits it to the local server through the network for download and extraction by users.

[0003] It can be seen that the video processing process in the cloud includes three steps: rendering, encoding, and transmission. Since the rendered content downloaded and extracted by the user is compressed by encoding and transmission, the content displayed on the user terminal after encoding and transmission may be different from the rendered content, which wastes rendering computing power and affects the user's rendering experience. Summary of the invention

[0004] The present application provides a video processing method, system and related devices, through which the quality of data downloaded and extracted by users can be improved, the consistency of the rendered content downloaded and extracted by users and the data obtained by rendering in the cloud can be ensured, and the user's rendering experience can be improved.

[0005] In a first aspect, the present application provides a video processing method, which is applied to a video processing device, wherein the video processing device is arranged in at least one data center, and a communication connection is established between the at least one data center and a terminal device. The method may include:

[0006] Acquire three-dimensional model data and network status parameters, wherein the network status parameters are parameters related to the network status of the communication connection, and the network status parameters include one or more of the following: bandwidth, packet loss rate, delay and jitter;

[0007] Determine rendering parameters according to the network status parameters, wherein the rendering parameters include one or more of the following: frame rate, resolution, and rendering layer status;

[0008] Rendering the three-dimensional model data based on the rendering parameters to generate target video data, wherein the target video data includes at least one rendering layer;

[0009] Encoding the target video data to obtain a target video code stream;

[0010] The target video code stream is sent to the terminal device via the communication connection.

[0011] It is understandable that without considering the network status parameters, the high-quality content rendered cannot be transmitted to the terminal device due to the poor network quality, so that the high-quality target video data cannot be displayed in the terminal device, thereby wasting rendering computing power. In this application, the video processing device takes into account the network status parameters at this time when rendering, that is, the current network status parameters can carry the target video data rendered, so the target video data can be displayed in the terminal device without loss, thereby improving the user's rendering experience and not wasting rendering computing power.

[0012] In a possible implementation of the first aspect, determining the rendering parameter according to the network status parameter includes:

[0013] According to the matching degree between the network state parameter and a plurality of preset network states, determining the target video processing strategy with the highest matching degree from a plurality of video processing strategies in a video processing strategy library, wherein the video processing strategy library stores a plurality of video processing strategies corresponding to a plurality of preset network states, the preset network state includes at least one of the network state parameters, and each of the plurality of video processing strategies includes the rendering parameter;

[0014] The rendering parameters are determined based on the target video processing strategy.

[0015] It can be seen that the video processing device pre-stores video processing strategies corresponding to different network status parameters, and then when executing the rendering task, the corresponding video processing strategy is selected according to the current network status parameters, and finally the rendering parameters are determined based on the video processing strategy. The rendering speed can be improved by pre-setting.

[0016] In a possible implementation manner of the first aspect, the target video processing strategy further includes encoding parameters and transmission parameters, and the method further includes:

[0017] Determining the encoding parameters and the transmission parameters based on the target video processing strategy;

[0018] The encoding of the target video data to obtain a target video code stream, and sending the target video code stream to the terminal device via the communication connection, comprises:

[0019] Encode the target video data based on the encoding parameters to obtain a target video code stream;

[0020] The target video code stream is sent to the terminal device via the communication connection based on the transmission parameters.

[0021] It can be seen that the target video processing strategy corresponding to the network status parameters also includes encoding parameters and transmission parameters. Therefore, after determining the current network status parameters, rendering, encoding and transmission can be performed in a pre-set manner, thereby improving the speed of cloud video processing.

[0022] In a possible implementation of the first aspect, determining the rendering parameter according to the network status parameter includes:

[0023] Calculate the target information entropy contained in the target video data according to the network state parameter;

[0024] The rendering parameters are determined based on the target information entropy.

[0025] It can be seen that the entropy of the rendered content is constrained according to the network state parameters, thereby determining the target information entropy. Therefore, the content rendered is obtained with the target information entropy as the target, so the above rendered content is transmitted without loss under the current network state parameters.

[0026] In a possible implementation manner of the first aspect, the calculating, according to the network state parameter, a target information entropy contained in the target video data includes:

[0027] Calculate the information entropy interval contained in the target video bitstream according to the network state parameter;

[0028] The target information entropy included in the target video data is determined according to the information entropy interval.

[0029] It is understandable that, under the premise that the network state parameters are determined, the encoder can perform deterministic encoding to obtain the target video bitstream, thereby providing a deterministic information entropy interval. Therefore, the information entropy determined from the deterministic information entropy interval is also determined, and the rendering content and rendering parameters are adjusted with the information entropy output as the target, so as to ensure that the content rendered with the rendering parameters can be transmitted under the current network state parameters.

[0030] In a possible implementation of the first aspect, determining the rendering parameter according to the network status parameter includes:

[0031] The rendering parameters are determined through a neural network according to the network state parameters.

[0032] It can be seen that under any network state parameters, the rendering parameters under the current network state parameters can be determined through the self-learning method of the neural network, thereby ensuring that the target video data rendered under the network state parameters can be displayed in the terminal device without loss.

[0033] In a possible implementation of the first aspect, the method further includes:

[0034] Determining encoding parameters and transmission parameters through the neural network according to the network state parameters;

[0035] The encoding of the target video data to obtain a target video code stream, and sending the target video code stream to the terminal device via the communication connection, comprises:

[0036] Encode the target video data based on the encoding parameters to obtain a target video code stream;

[0037] The target video code stream is sent to the terminal device via the communication connection based on the transmission parameters.

[0038] It can be seen that under any network status parameters, the encoding parameters and transmission parameters under the current network status parameters can be determined through the self-learning method of the neural network. It can be applied to different network environments, automatically and accurately tuned, and determine the rendering, transmission and encoding that meet the current network status parameters, so that the encoded target video code stream can be transmitted to the terminal device without loss.

[0039] In a possible implementation of the first aspect, the rendering layer state includes rendering layer on and rendering layer off, and determining the rendering parameter according to the network state parameter includes:

[0040] Determine that the network status is normal according to the network status parameter, and perform one of the following operations:

[0041] The frame rate is increased, the resolution is increased, and a target rendering layer in the at least one rendering layer is enabled according to the rendering layer state.

[0042] It can be seen that when the network state indicated by the current network state parameter is normal, it means that the rendered high-quality video data can be sent to the terminal device through the network, so the rendering quality can be improved to obtain high-quality video data.

[0043] In a possible implementation of the first aspect, the rendering layer state includes rendering layer on and rendering layer off, and determining the rendering parameter according to the network state parameter includes:

[0044] Determine that the network status is abnormal according to the network status parameter, and perform one of the following operations:

[0045] The frame rate is reduced, the resolution is reduced, and a target rendering layer in the at least one rendering layer is closed according to the rendering layer state.

[0046] It can be seen that when the network status indicated by the current network status parameters is abnormal, it means that if the high-quality video data rendered according to the existing rendering computing power cannot be sent to the device in the network, the rendering quality can be reduced to avoid wasting rendering computing power and encoding computing power.

[0047] In a second aspect, the present application provides a video processing device, wherein the video processing device is arranged in at least one data center, a communication connection is established between the at least one data center and a terminal device, and the video processing device includes a communication module and a processing module, wherein:

[0048] The communication module is used to obtain the three-dimensional model data and network status parameters, where the network status parameters are parameters related to the network status of the communication connection, and the network status parameters include one or more of the following: bandwidth, packet loss rate, delay and jitter;

[0049] The processing module is used to determine rendering parameters according to the network status parameters, wherein the rendering parameters include one or more of the following: frame rate, resolution, and rendering layer status;

[0050] The processing module is further used to render the three-dimensional model data based on the rendering parameters to generate target video data, wherein the target video data includes at least one rendering layer;

[0051] The processing module is further used to encode the target video data to obtain a target video code stream;

[0052] The communication module is further used to send the target video code stream to the terminal device via the communication connection.

[0053] In a possible implementation manner of the second aspect, the processing module is specifically configured to:

[0054] According to the matching degree between the network state parameter and a plurality of preset network states, determining the target video processing strategy with the highest matching degree from a plurality of video processing strategies in a video processing strategy library, wherein the video processing strategy library stores a plurality of video processing strategies corresponding to a plurality of preset network states, the preset network state includes at least one of the network state parameters, and each of the plurality of video processing strategies includes the rendering parameter;

[0055] The rendering parameters are determined based on the target video processing strategy.

[0056] In a possible implementation manner of the second aspect, the processing module is further used to determine the encoding parameter and the transmission parameter based on the target video processing strategy;

[0057] The processing module is specifically used to encode the target video data based on the encoding parameters to obtain a target video code stream;

[0058] The communication module is specifically used to: send the target video code stream to the terminal device via the communication connection based on the transmission parameters.

[0059] In a possible implementation of the second aspect, the device further includes a renderer, and the processing module is specifically configured to:

[0060] Calculate the target information entropy contained in the target video data according to the network state parameter;

[0061] The rendering parameters are determined based on the target information entropy.

[0062] In a possible implementation manner of the second aspect, the processing module is specifically configured to:

[0063] Calculate the information entropy interval contained in the target video bitstream according to the network state parameter;

[0064] The target information entropy contained in the target video data is determined according to the information entropy interval.

[0065] In a possible implementation manner of the second aspect, the processing module is specifically configured to:

[0066] The rendering parameters are determined through a neural network according to the network state parameters.

[0067] In a possible implementation manner of the second aspect, the processing module is further used to determine a coding parameter and a transmission parameter through the neural network according to the network state parameter;

[0068] The processing module is specifically used to encode the target video data based on the encoding parameters to obtain a target video code stream;

[0069] The communication module is specifically configured to send the target video code stream to the terminal device via the communication connection based on the transmission parameters.

[0070] In a possible implementation of the second aspect, the rendering layer state includes rendering layer on and rendering layer off, and the processing module is specifically configured to:

[0071] Determine that the network status is normal according to the network status parameter, and perform one of the following operations:

[0072] The frame rate is increased, the resolution is increased, and a target rendering layer in the at least one rendering layer is enabled according to the rendering layer state.

[0073] In a possible implementation of the second aspect, the rendering layer state includes rendering layer on and rendering layer off, and the processing module is specifically configured to:

[0074] Determine that the network status is abnormal according to the network status parameter, and perform one of the following operations:

[0075] The frame rate is reduced, the resolution is reduced, and a target rendering layer in the at least one rendering layer is closed according to the rendering layer state.

[0076] In a third aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory; the processor is used to execute instructions stored in the memory so that the computing device implements the method described in any one of the first aspects above.

[0077] Optionally, the computing device further comprises a communication interface, wherein the communication interface is used to receive and / or send data, and / or the communication interface is used to provide input and / or output for the processor.

[0078] It should be noted that the above embodiment is described by taking a processor (or general-purpose processor) that executes the method by calling a computer specification as an example. In the specific implementation process, the processor can also be a dedicated processor, in which case the computer instructions have been pre-loaded in the processor. Optionally, the processor can also include both a dedicated processor and a general-purpose processor.

[0079] Optionally, the processor and the memory may also be integrated into one device, that is, the processor and the memory may also be integrated together.

[0080] In a fourth aspect, an embodiment of the present application further provides a computing device cluster, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory;

[0081] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in any one of the first aspects.

[0082] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on at least one processor, the method described in any one of the first aspects is implemented.

[0083] In a sixth aspect, the present application provides a computer program product, which includes computer instructions. When the instructions are executed on at least one processor, the method described in any one of the first aspects is implemented.

[0084] Optionally, the computer program product may be a software installation package or an image package. When the aforementioned method is required, the computer program product may be downloaded and executed on a computing device.

[0085] The beneficial effects of the technical solutions provided in the second to sixth aspects of the present application can refer to the beneficial effects of the technical solution of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a schematic diagram of a cloud rendering optimization process provided by an embodiment of the present application;

[0087] Figure 2A It is a schematic diagram of the architecture of a cloud rendering system provided in an embodiment of the present application;

[0088] Figure 2B is a schematic diagram of the architecture of another cloud rendering system provided in an embodiment of the present application;

[0089] Figure 3 is a schematic diagram of the architecture of a video processing system provided by an embodiment of the present application;

[0090] Figure 4 is a schematic diagram of another video processing system provided in an embodiment of the present application;

[0091] Figure 5 is a schematic diagram of another video processing system provided in an embodiment of the present application;

[0092] Figure 6 is a schematic diagram of another video processing system provided in an embodiment of the present application;

[0093] Figure 7 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0094] Figure 8 is a structural schematic diagram of a video processing device 80 provided in an embodiment of the present application;

[0095] Fig. 9 The figure is a schematic diagram of the structure of a computing device 90 provided in an embodiment of the present application;

[0096] Fig.10 Shown is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0097] Fig.11 Shown is a schematic diagram of one or more computing devices in a computing device cluster provided by an embodiment of the present application being connected through a network. DETAILED DESCRIPTION

[0098] The embodiments of the present invention are described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs, and the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0099] It is understandable that traditional rendering is based on the graphics processing unit (GPU) of the local device, but the GPU of the local device can only process limited data and cannot effectively complete complex scenes and large-scale rendering tasks. Cloud rendering is achieved by submitting rendering tasks to the service cluster in the cloud. In cloud rendering, the rendering tasks and related scene files are uploaded to the cloud rendering platform, and then the cloud rendering platform can assign the rendering tasks to multiple servers in the cloud for rendering, and encode the rendered images and send them to the client, so that the client can decode the received image data, obtain the rendered images and display them.

[0100] See also Figure 1 , Figure 1 Schematic diagram of a cloud rendering optimization process provided by an embodiment of the present application. Figure 1 It can be seen that the cloud rendering platform 130 includes a rendering module 101, an encoding module 102 and a transmission module 103. The cloud rendering platform 130 is connected to the terminal device 110 through a network. The cloud rendering optimization process of the cloud rendering platform 130 involves multiple links, mainly including one or more of the following: the rendering process is adjusted by the rendering module 101, the encoding process is adjusted by the encoding module 102, and the transmission process is adjusted by the transmission module 103, etc. The above links can directly affect the user experience. In actual use, the above three links can be adjusted in a targeted manner. For example:

[0101] like Figure 1 As shown, the adjustment of the rendering module 101 in the rendering link by the cloud rendering platform 130 is usually achieved in combination with layered rendering. There are mainly the following layering methods for layered rendering. The first is to roughly layer according to the content of the rendering, such as dividing it into a foreground layer, a mid-ground layer, and a background layer according to the scene, etc. The second is to finely layer according to the visual attributes of the object, such as dividing it into a color layer, a highlight layer, a shadow layer, a reflection layer, a refraction layer, and a luminous layer, etc. Layered rendering greatly improves the flexibility of rendering. The cloud server can use appropriate rendering strategies according to the hardware capabilities, and can also perform differentiated control. In addition, the adjustment of the rendering module 101 in this link can also include adjusting rendering parameters, such as frame rate, resolution, etc.

[0102] The cloud rendering platform 130 may adjust the encoding module 102 in the encoding process, including the code control mode (such as constant bit rate, variable bit rate, constant quality, etc.), encoding bit rate, encoding quality, encoding standard, etc. Figure 1 As shown, the cloud rendering platform 130 can also use layered coding in the coding module 102 for encoding. Layered coding, also known as scalable video coding, divides the video in time, space and quality, and outputs a multi-layer code stream (usually including a base layer and multiple enhancement layers). Among them, the data of the base layer can be used to decode the basic video content, which can ensure that the user can watch smooth images when the channel is limited or the channel environment is complex. When the channel environment is good or the channel resources are abundant, the data of the enhancement layer is used to improve the video quality.

[0103] like Figure 1 As shown, the cloud rendering platform 130 may adjust the transmission module 103 in the transmission link, including adjusting the adopted transmission algorithm, for example, Web real-time communications (WebRTC), and adjusting the transmission parameters in the transmission algorithm, etc. In addition, the concept of 5G slicing is introduced in the fifth generation mobile communication technology (5G). The 5G standard slices the network channel, for example, including 5G slice A and 5G slice B. Different 5G slices correspond to different application scenarios, such as high-definition live broadcast, games, electricity, industry, medical care, etc. Cloud rendering applications can achieve a good balance between cost and experience by selecting appropriate slices for transmission. For example, for the rendering task of a game application, the slice corresponding to the game is selected for transmission.

[0104] from Figure 1 It can be seen that in the rendering module 101, the cloud rendering platform 130 can determine the rendering scheme in advance based on hardware capabilities and cost control, and maintain the rendering scheme unchanged regardless of the network status. The tuning of the cloud rendering application is mainly based on the network status to adjust the two links of the encoding module 102 and the transmission module 103. For the encoding module 102, it is usually to adjust the number of base layers and enhancement layers, or adjust the encoding parameters, etc. For the transmission module 103, it is usually to adjust the adopted transmission algorithm or adjust the transmission parameters, etc.

[0105] It can be seen that Figure 1 The cloud rendering platform 130 shown does not take the network status into consideration during the rendering process according to the determined rendering scheme, which may cause the following problems: wasting computing power and reducing the experience.

[0106] Problem 1: Waste of computing power. Regardless of the network status, consistent rendering according to the rendering scheme may result in the inability to transmit the rendered high-quality video to the user when the network quality is poor, thereby wasting rendering computing power. In addition, since the higher the image quality, the greater the computing power overhead of the encoding module 102, high-quality rendering also causes a waste of encoding computing power.

[0107] Problem 2: Reduced experience. Regardless of the network status, consistent rendering is performed according to the rendering scheme. This may result in the encoding module 102 deciding the information to be transmitted to the user when the network quality is poor. Since the encoding module 102 cannot accurately perceive the rendering content, incorrect choices during encoding will result in low image quality, thereby affecting the user experience.

[0108] In summary, the current optimization of cloud rendering is mainly based on the linkage of encoding and transmission based on network status. Rendering generally only considers hardware capabilities and cost control, and lacks linkage with subsequent links such as encoding and transmission. This has, to a certain extent, caused a decline in user experience and an increase in costs.

[0109] In view of this, the present application provides a cloud video processing method, a cloud video processing system and related devices, which are applied to a cloud rendering platform. In the embodiment of the present application, a rendering link is added to the tuning process of cloud rendering, and different content or different image quality is provided through rendering according to different network status parameters, and then the whole link linkage tuning is performed with downstream encoding and transmission to improve user experience and save costs.

[0110] See also Figure 2A , Figure 2A 1 is a schematic diagram of the architecture of a cloud rendering system provided by an embodiment of the present application. The cloud rendering system of the present application is used to render the original model data such as the 3D model of the virtual scene to obtain an image or video, that is, image rendering or video rendering. The cloud rendering system of the present application may include: one or more terminal devices 110, a network device 120, and a cloud rendering platform 130. The cloud rendering platform 130 can be specifically deployed in one or more data centers of a public cloud, or in one or more data centers of a private cloud. The terminal device 110 and the cloud rendering platform 130 are generally deployed in different data centers.

[0111] The terminal device 110 may be a device that needs to display rendered images in real time, such as a virtual reality device (VR) for flight training, a computer for virtual games, and a smart phone for displaying virtual scenes, etc., which are not specifically limited here. The device in may be a high-configuration, high-performance device (e.g., multi-core, high main frequency, large memory, etc.), or a low-configuration, low-performance device (e.g., single-core, low main frequency, small memory, etc.). In some specific embodiments, the terminal device 110 may include hardware, an operating system, and a rendering application client.

[0112] The network device 120 is used to transmit data between the terminal device 110 and the cloud rendering platform 130 through any communication mechanism / communication standard communication connection. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0113] The cloud rendering platform 130 includes one or more rendering nodes, each of which includes rendering hardware, virtualization services, rendering engines, and rendering application servers from top to bottom. Among them, the rendering hardware includes computing resources, storage resources, and network resources. The computing resources can adopt a heterogeneous computing architecture, for example, a central processing unit (CPU) + graphics processing unit (GPU) architecture, a CPU + AI chip, a CPU + GPU + AI chip architecture, etc., which are not specifically limited here. Storage resources may include storage devices such as memory and video memory. Network resources may include network cards, port resources, address resources, etc. Virtualization services virtualize the resources of rendering nodes into vCPUs, etc. through virtualization technology, and flexibly isolate independent resources according to user needs to run user applications. Services. Commonly, virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM, BMS, and containers can run rendering engines and rendering application servers. The rendering engine is used to implement rendering algorithms. The rendering application server is used to call the rendering engine to complete the rendering of the rendered image.

[0114] In one implementation, one or more rendering nodes in the cloud rendering platform 130 obtain the original model data such as the 3D model of the virtual scene and the network state parameters of the above-mentioned communication connection, wherein the network state parameters are parameters related to the network state of the communication connection, and the network state parameters include one or more of the following: bandwidth, packet loss rate, delay and jitter. Then, one or more rendering nodes in the cloud rendering platform 130 determine rendering parameters according to the network state parameters, and render the original model data such as the 3D model of the above-mentioned virtual scene based on the rendering parameters to generate target video data. Among them, the rendering parameters include one or more of the following: frame rate, resolution and rendering layer state, and the target video data includes at least one rendering layer. Finally, one or more rendering nodes in the cloud rendering platform 130 encode the target video data to obtain a target video code stream, and send the target video code stream to the terminal device 110 via the communication connection.

[0115] The rendering application client on the terminal device 110 and the rendering application server of the cloud rendering platform 130 are collectively referred to as rendering applications. Common rendering applications may include: game applications, VR applications, movie special effects, and animations, etc. The user inputs an operation instruction through the rendering application client, and the rendering application client sends the operation instruction to the rendering application server. The rendering application server calls the rendering engine to generate rendering content and sends the rendering content to the rendering application client. The rendering application client then converts the rendering content into an image and presents it to the user.

[0116] In a specific implementation, the rendering application server and the rendering application client may be provided by the rendering application provider, and the rendering engine may be provided by the cloud service provider. For example, the rendering application may be a game application, and the game developer of the game application installs the game application server on the remote rendering platform provided by the cloud service provider. The game developer of the game application provides the game application client to the user for download via the Internet and installs it on the user's terminal device. In addition, the cloud service provider also provides a rendering engine, which can provide computing power for the game application. In another specific implementation, the rendering application client, the rendering application server, and the rendering engine may all be provided by the cloud service provider.

[0117] exist Figure 2B The cloud rendering system shown also includes a management device 140. The management device 140 can be a device provided by a third party other than the user's terminal device and the cloud rendering platform 130 of the cloud service provider. For example, the management device 140 can be a device provided by a game developer. The game developer can manage the rendering application through the management device 140. It can be understood that the management device 140 can be set on the remote rendering platform or outside the cloud rendering platform, which is not specifically limited here.

[0118] See also Figure 3 , Figure 3 is a schematic diagram of the architecture of a video processing system provided in an embodiment of the present application, Figure 3 The video processing system shown can be Figure 2A or Figure 2B The cloud rendering system shown in Figure 1 is a manifestation of the video processing scenario. Figure 3 It can be seen that the video processing system includes a cloud rendering platform 130 and a terminal device 110. The cloud rendering platform 130 may include a network detection module 132 and a control module 131. For a description of the cloud rendering platform 130, please refer to Figure 2A , which will not be described here. Among them:

[0119] The network detection module 132 is a software module for detecting and probing the network status. It can obtain various parameters and indicators of the network status by sending network requests, receiving network responses and analyzing network data.

[0120] In one implementation, the network detection module 132 can be used to identify network failures, measure network delays, detect network packet loss rates, monitor network bandwidth, etc. It can help network administrators or system administrators monitor and manage the operating status of the network and promptly discover and solve network problems. For example, the network detection module 132 generally has the following functions: 1) Sending network requests: By sending network requests, the network detection module can test the reachability and connectivity of the network. 2) Receiving network responses: The network detection module can receive response information returned by network devices, such as the response time of the ping command. 3) Analyzing network data: The network detection module can analyze and parse the received network data to obtain network performance indicators, such as delay, packet loss rate, etc. 4) Monitoring network status: By regularly sending network requests and receiving responses, the network detection module can monitor the status changes of the network in real time.

[0121] Combination Figure 2A or Figure 2B It can be seen that the network detection module 132 can detect and detect the network status of the network device 120. Because the communication connection between the terminal device 110 and the cloud rendering platform 130 is established through the network device 120, Figure 3 The network detection module 132 shown can detect and detect the network state of the above-mentioned communication connection, thereby obtaining the network state parameters of the communication connection. For example, for a 5G connection, the detected and detected network state parameters can specifically be the type of 5G slice and the network state parameters corresponding to the 5G slice. Then, the network detection module 132 feeds back the detected network state parameters to the control module 131.

[0122] The control module 131 may be a module for controlling the rendering module 101 to perform a rendering process. After receiving the network status parameters from the network detection module 132 , the rendering parameters in the rendering module 101 may be determined according to the network status parameters.

[0123] It is understandable that the rendering module 101 can be hierarchical (for example, the rendering module 101 can perform layered rendering on the original model data generated by the rendering application to obtain a rendering result including multiple rendering layers). At this time, the rendering parameters include rendering parameters corresponding to different rendering layers. Since the computing power required for the cloud rendering platform 130 to render different rendering layers is different, the control module 131 can determine the rendering parameters corresponding to different rendering layers that match the network state parameters according to the network state parameters (for example, the rendering parameters may include one or more of the following: frame rate, resolution, and rendering layer state, etc.), so that the rendered content can be transmitted through the communication connection corresponding to the network state parameters. This can save rendering computing power and avoid the inability to transmit high-quality content rendered with more computing power due to poor network quality.

[0124] Exemplarily, when graded according to scene categories, the rendering layers include: one or more of foreground layer, midground layer, background layer and image blocks; when graded according to visual attributes, the rendering layers include: one or more of color layer, highlight layer, shadow layer, reflection layer, refraction layer and luminous layer; when graded according to rendering technology, the rendering layers include: one or more of depth of field, lighting, ground reflection and dynamic light source shadow.

[0125] In one implementation, the control module 131 can also be a module that controls the encoding module 102 to perform encoding and the transmission module 103 to perform transmission. After receiving the network status parameters from the network detection module 132, the encoding parameters required by the encoding module 102 for encoding and the transmission parameters required by the transmission module 103 for transmission can be determined according to the network status parameters.

[0126] It is understandable that the coding layer can be hierarchical (for example, the coding module 102 uses a layered coding method to encode the rendering result generated by the rendering module 101, and the video code stream generated after the layered coding contains multiple coding layers), and the coding parameters include coding parameters corresponding to different coding layers. Since the computing power required for the cloud rendering platform 130 to encode different coding layers is different, the control module 131 can determine the coding parameters corresponding to different coding layers that match the network state parameters according to the network state parameters, so that the encoded content can be transmitted through the communication connection corresponding to the network state parameters. In one implementation, the coding layers can be graded at the levels of time, space, and quality, so the coding parameters include one or more of the coding parameters corresponding to time, the coding parameters corresponding to space, and the coding parameters corresponding to quality. Among them, the coding parameters include code control methods (such as constant bit rate, variable bit rate, constant quality, etc.), coding bit rate, coding image quality, and coding standards.

[0127] In addition, the transmission module 103 needs to use a variety of transmission algorithms and corresponding transmission parameters to transmit the video code stream generated by the encoding module 102, so the transmission parameters include transmission parameters corresponding to different transmission algorithms and different transmission parameters. Therefore, the control module 131 can determine the transmission algorithm and the corresponding transmission parameters that match the network state parameters according to the network state parameters, so that the rendered encoded content can be transmitted through the communication connection corresponding to the network. Further, in 5G, different transmission algorithms and different transmission parameters can be combined with different 5G slices, so the transmission parameters can include transmission parameters corresponding to different 5G slices combined with different transmission algorithms and different transmission parameters. Among them. The transmission parameters include network slices, etc.

[0128] When the cloud rendering platform 130 performs rendering in the rendering module 101, the 3D model data may be rendered based on the rendering parameters determined by the control module 131 according to the network status parameters to generate target video data, wherein the target video data includes at least one rendering layer.

[0129] In one implementation, in the rendering module 101, since rendering is hierarchical, the rendering parameters include rendering parameters corresponding to different rendering layers matched with the network state parameters. The cloud rendering platform 130 can render different levels of the video content according to the rendering parameters corresponding to different rendering layers, and finally obtain cloud rendering content.

[0130] When the cloud rendering platform 130 performs encoding through the encoding module 102 , the target video data may also be encoded according to the encoding parameters determined by the control module 131 according to the network status parameters to obtain a target video code stream.

[0131] In one implementation, during the encoding process, since the encoding is hierarchical, the encoding parameters include encoding parameters corresponding to different encoding layers that match the network state parameters. The cloud rendering platform 130 can encode the target video data at different levels according to the encoding parameters corresponding to different encoding layers based on the network state parameters, and finally obtain the target video code stream.

[0132] During the transmission process performed by the transmission module 103 , the cloud rendering platform 130 may also send the target video code stream to the terminal device via the communication connection according to the transmission parameters determined by the control module 131 according to the network status parameters.

[0133] In one implementation, during the transmission process, multiple transmission algorithms and corresponding transmission parameters are included, so the cloud rendering platform 130 can adjust the adopted transmission algorithm and its corresponding transmission parameters based on the network status parameters to transmit the code stream of the cloud rendering content.

[0134] Therefore, if Figure 3 As shown, in the optimization process of cloud rendering, the three links of rendering module 101, encoding module 102 and transmission module 103 are linked. The cloud rendering platform 130 renders and provides different image qualities according to different network states, and then performs linkage optimization with encoding and transmission.

[0135] See also Figure 4 , Figure 4 It is a schematic diagram of another video processing system provided in an embodiment of the present application. Figure 4 The video processing system shown can be Figure 2A or Figure 2B The cloud rendering system shown in Figure 1 is another manifestation of the video processing scenario. Figure 4 It can be seen that the video processing system includes a cloud rendering platform 130 and a terminal device 110. The cloud rendering platform 130 may include a network detection module 132, a control module 131 and a video processing strategy library 133. For a description of the cloud rendering platform 130, please refer to Figure 2A , which will not be described here. Among them:

[0136] For the communication connection between the cloud rendering platform 130 and the terminal device 110, and the related description of the network detection module 132 and the control module 131, please refer to Figure 3 , I will not go into details here.

[0137] The video processing strategy library 133 stores multiple video processing strategies corresponding to multiple preset network states. For example, the video processing strategy library A = {(s1, a1), (s2, a2), (s3, a3), .......}, s represents the preset network state, a represents the video processing strategy, that is, one preset network state corresponds to one video processing strategy.

[0138] Among them, the preset network status includes one or more network status parameters of the network device in the process of transmitting data that are pre-acquired. The network status parameters can be key performance indicators, and the key performance indicators include but are not limited to: packet loss rate, bandwidth, latency, total packet loss, time jitter, round-trip time, effective rate of payload data, average rate of payload data, maximum value of the second rate of payload data, minimum value of the second rate of payload data, total number of packets, and effective transmission time of payload data.

[0139] It can be understood that the video processing strategy in the video processing strategy library 133 is a parameter for cloud rendering processing obtained by analyzing, learning and adjusting the data transmitted under different network state parameters. Therefore, the content transmitted under different network state parameters is analyzed and learned to obtain the transmission algorithm used to transmit the content and the corresponding transmission parameters. Because the transmitted content is encoded, the encoding parameters used when encoding the content can be obtained by analyzing and learning the transmitted content. Because the rendered content is encoded, and the network state parameters are considered during encoding, there is lossless encoding and / or lossy encoding of the rendered content. Therefore, the encoded data is mapped to the rendering layer, and the rendering parameters corresponding to the lossless encoding and / or the rendering parameters corresponding to the lossy encoding can be reversely deduced under the consideration of the network state parameters. Thus, the rendering parameters, encoding parameters and transmission parameters corresponding to the network state parameters can be determined under a certain network state parameter. For example, the preset network state s1 corresponds to the video processing strategy 1, and the video processing strategy 1 includes the rendering parameters of the ordinary background layer in multiple rendering layers, the encoding parameters of the base layer in multiple encoding layers, and the transmission parameters corresponding to the game slice. The preset network state s2 corresponds to video processing strategy 2. Video processing strategy 1 includes rendering parameters of global illumination, depth of field, and shadows, encoding parameters of the enhancement layer, and transmission parameters corresponding to game slices. The preset network state s3 corresponds to video processing strategy 3. Video processing strategy 3 includes rendering parameters of ordinary background, encoding parameters of the enhancement layer, and transmission parameters corresponding to live slices. The preset network state s4 corresponds to video processing strategy 4. Video processing strategy 4 includes rendering parameters of animation special effects, encoding parameters of the base layer, and transmission parameters corresponding to ordinary slices.

[0140] In actual scenarios, such as Figure 4 As shown, after receiving the user request ( Figure 4 (not shown), obtain the three-dimensional model data and the network state parameter s of the network device 120 detected by the network detection module 132, wherein the network state parameter s includes one or more of the above network state parameters, that is, Figure 2A or Figure 2BThe network state parameter s of the communication connection between the cloud rendering platform 130 and the terminal device 110 is shown. The network detection module 132 transmits the above network state parameter s to the control module 131. Since the network state parameter s is a parameter related to the preset network state, the control module 131 can determine the preset network state si that is most similar to the network state parameter s (also known as the highest matching degree) from the video processing strategy library A, where i is a positive integer greater than or equal to 1.

[0141] It should be noted that "most similar or with the highest matching degree" means that the key performance indicators included in the real-time detected network state parameter s are the same as or similar to the key performance indicators included in the preset network state si stored in the video processing strategy library A. For example, the similarity value between the network state parameter s and the corresponding network state parameter in the preset network state si can be calculated according to the difference between the network state parameter s and the preset network state si. When the similarity value is less than a threshold preset by the system, the network state si is determined to be the network state that is most similar to the network state parameter s.

[0142] Therefore, the control module 131 determines the video processing strategy ai corresponding to the preset network state si from the video processing strategy library A according to the preset network state si, and uses the video processing strategy ai as the video processing strategy a finally used. The video processing strategy a includes rendering parameters. The cloud rendering platform 130 can render the three-dimensional model data based on the rendering parameters through the rendering module 101 to generate target video data, and the target video data includes at least one rendering layer (for example, rendering layer 1 and / or rendering layer 2).

[0143] In one implementation, the video processing strategy a also includes encoding parameters and transmission parameters. The cloud rendering platform 130 can determine the encoding parameters and transmission parameters based on the video processing strategy a. Therefore, the cloud rendering platform 130 can encode the target video data based on the encoding parameters through the encoding module 102 to obtain the target video code stream. Exemplarily, since the encoding is hierarchical, the encoding parameters include encoding parameters corresponding to different encoding layers (such as the base layer and / or the enhancement layer) matched with the network state parameters.

[0144] In one implementation, the cloud rendering platform 130 may send the target video code stream to the terminal device via a communication connection based on the transmission parameters through the transmission module 103 .

[0145] It can be seen that the above-mentioned rendered content, encoded content and transmitted content are determined based on the network status parameters at that time, so the rendered content can be correctly encoded without affecting the content quality, so that the rendered content can be displayed on the terminal device.

[0146] See also Figure 5 , Figure 5It is a schematic diagram of another video processing system provided in an embodiment of the present application. Figure 5 The video processing system shown can be Figure 2A or Figure 2B The cloud rendering system shown in Figure 1 is another manifestation of the video processing scenario. Figure 5 It can be seen that the video processing system includes a cloud rendering platform 130 and a terminal device 110. The cloud rendering platform 130 may include a network detection module 132 and a control module 131. For a description of the cloud rendering platform 130, please refer to Figure 2A , which will not be described here. Among them:

[0147] For the communication connection between the cloud rendering platform 130 and the terminal device 110, the relevant description of the network detection module 132 and the control module 131 can be referred to Figure 3 , I will not go into details here.

[0148] like Figure 5 As shown, after receiving the user request ( Figure 5 The cloud rendering platform 130 renders the three-dimensional model data based on the rendering parameters through the rendering module 101 to generate the target video data, and encodes the target video data through the encoding module 102 to obtain the target video code stream. Among them, the target video data includes at least one rendering layer. Under the above network state parameters, the cloud rendering platform 130 sends the target video code stream to the terminal device through the transmission module 103 via a communication connection.

[0149] It is understandable that the target information entropy refers to the information entropy expected to be generated during the rendering process under the network state parameters at this time. During the rendering process, the image is finally generated by tracing and calculating the light in the scene. In this process, the propagation path of light in the scene is very complex and will be affected by many factors such as material, lighting, and shadow. Information entropy is a metric used to measure the uncertainty or amount of information in a system, indicating the degree of uneven distribution of information diversity in the system. In rendering, information entropy can be used to describe the noise and detail level in the image. It is understandable that higher information entropy means higher image quality and more noise and details in the image. In the rendering process, higher computing power is required, so a matching network state is required for lossless encoding and lossless transmission. Lower information entropy means lower image quality and less noise and details in the image. In the rendering process, higher computing power is not required, so lossless encoding and lossless transmission can be achieved when the network state is not good.

[0150] In one implementation, the cloud rendering platform 130 also includes an encoder ( Figure 5 (not shown), since the network state parameters detected by the network detection module 132 are deterministic networks, the encoder in the encoding module 102 can perform deterministic encoding under the premise that the network state parameters such as frame rate, delay and output bandwidth are determined, that is, the control module 131 can calculate the information entropy interval contained in the target video code stream output by the encoder in the encoding module 102 according to the network state parameters, and then determine the target information entropy contained in the rendering content (that is, target video data) output by the renderer from the information entropy interval. Therefore, in the case of a deterministic network, deterministic encoding can be performed, and deterministic information entropy can be obtained from the information entropy interval obtained by deterministic encoding. Finally, the control module 131 can determine the rendering parameters with the deterministic information entropy (that is, the target information entropy) output as the target, thereby adjusting the rendering content according to the rendering parameters.

[0151] Exemplarily, in the rendering module 101, multi-level rendering output and combination determination of rendering parameters are implemented based on the target information entropy. For example, based on the target information entropy, the entropy renderer A renders the contents of the main viewing angle 10m range and inner blocks 1-10, based on the target information entropy, the entropy renderer B selects the contents of the main viewing angle 10m-50m range and inner blocks 100-500, and based on the target information entropy, the ordinary renderer C renders other contents. For another example, based on the target information entropy, the entropy renderer A renders the foreground digital human action, based on the target information entropy, the entropy renderer B renders the background, and based on the target information entropy, the ordinary renderer C renders the depth of field.

[0152] In one implementation, the control module 131 may use a neural rendering method to determine the target information entropy. Generally speaking, neural rendering trains a neural network-based plenoptic function F Θ , to achieve view rendering at any position and viewing angle.

[0153] F Θ (x,y,z,θ,φ)=(RGBσ)

[0154] In order to achieve deterministic entropy rendering, the neural rendering equation can be expanded by adding the network state w during training and inference, thereby achieving deterministic entropy rendering, that is, the target information entropy includes (RGBσ).

[0155] F Θ (x,y,z,θ,φ,w)=(RGBσ)

[0156] Among them, (x, y, z) represents the position, (θ, φ) represents the direction (polar coordinates), σ represents a single pixel point, and (RGBσ) represents the pixel value of a single pixel point.

[0157] Therefore, when the renderer can output deterministic information entropy, the encoding and transmission in the rendering process do not need to be dynamically adjusted. In this way, the multi-level linkage can be reduced to the rendering layer module, thereby achieving stable rendering quality.

[0158] See also Figure 6 , Figure 6 It is a schematic diagram of another video processing system provided in an embodiment of the present application. Figure 6 The video processing system shown can be Figure 2A or Figure 2B The cloud rendering system shown in Figure 1 is another manifestation of the video processing scenario. Figure 6 It can be seen that the video processing system includes a cloud rendering platform 130 and a terminal device 110. The cloud rendering platform 130 may include a network detection module 132 and a control module 131. For a description of the cloud rendering platform 130, please refer to Figure 2A , which will not be described here. Among them:

[0159] For the communication connection between the cloud rendering platform 130 and the terminal device 110, the relevant description of the network detection module 132 and the control module 131 can be referred to Figure 3 , I will not go into details here.

[0160] like Figure 6 As shown, after receiving the user request ( Figure 6(not shown in the figure), obtain the three-dimensional model data and the network status parameters of the network device 120 detected by the network detection module 132. The network detection module 132 transmits the above-mentioned network status parameters to the control module 131, and the control module 131 determines the rendering parameters (including frame rate, resolution and rendering layer status, etc.) through the neural network based on the above-mentioned network status parameters. Exemplarily, the control module 131 calculates the optimal value of the functional relationship determined by the neural network based on the network status parameters, and the parameters of the rendering layer corresponding to the optimal value are the rendering parameters. Among them, the functional relationship is used to indicate the functional relationship between the network status and the rendering module 101. Therefore, the cloud rendering platform 130 can render the three-dimensional model data corresponding to the user request based on the rendering layer parameters through the rendering module 101 to generate target video data, and the target video data includes at least one rendering layer.

[0161] In one implementation, the cloud rendering platform 130 determines the encoding parameters and the transmission parameters through a neural network according to the network status parameters, and then encodes the target video data based on the encoding parameters through the encoding module 102 to obtain the target video code stream; finally, the transmission module 103 sends the target video code stream to the terminal device via a communication connection based on the transmission parameters.

[0162] Exemplarily, in the rendering module 101, the renderer may adopt differentiable rendering based on neural rendering to determine differentiable rendering layer parameters. In the encoding module 102, the encoder may adopt differentiable encoding based on neural network to determine differentiable encoding layer parameters. In the transmission module 103, a transmission algorithm based on deep learning may be adopted to determine differentiable transmission layer parameters. Among them:

[0163] Differentiable rendering initially requires the following two resources: a reference image (ground truth) and initial scene parameters. The reference image is a two-dimensional target image, and the goal of differentiable rendering is to make the rendering result as close to this reference image as possible. The initial scene parameters can be a rough three-dimensional scene that is roughly close to the real scene, or a completely random three-dimensional scene. The differentiable rendering process includes: process A, the renderer renders an image based on the current scene; process B, compares the image rendered based on the initial scene with the ground truth to obtain a loss; process C, back-propagates the gradient obtained from this loss to the previous scene parameters, thereby continuously optimizing the scene parameters so that the rendered result is closer and closer to the ground truth; process D, continuously executes process A-process C until the rendered image is almost the same as the ground truth, and the corresponding scene parameters can be the differentiable rendering layer parameters.

[0164] Differentiable coding: The training of neural networks usually relies on the gradient descent algorithm. These non-differentiable codes (such as integer codes, permutation codes, etc.) cannot be directly applied to neural networks. Therefore, differentiable networks can be used to simulate these non-differentiable traditional codes. Specifically, a differentiable network structure can be designed, the traditional code is used as the input of the network, and the operation of the traditional code is simulated through the forward propagation process of the network. For example, suppose you want to simulate integer coding. Traditional integer coding maps an integer to a fixed-length binary vector, where each bit represents a certain bit of the integer. A differentiable network structure can be designed, the integer is used as the input, and the integer coding process is simulated through a series of fully connected layers, activation functions and other operations. Finally, the output of the network is the simulated integer code, which can be used as the parameter of the differentiable coding layer. Similarly, differentiable networks can be used to simulate other non-differentiable traditional codes, such as permutation codes, discrete codes, etc. By designing appropriate network structures and operations, these non-differentiable codes can be converted into differentiable forms, so that they can be applied to the training of neural networks. By converting traditional codes into differentiable forms, these codes can be flexibly applied in neural networks, thereby expanding the application scope of neural networks.

[0165] Transmission algorithms based on deep learning include: autoencoders, which are used for end-to-end learning of communication systems without channels. Among them, the autoencoder can use supervised learning and adopt a differentiable known channel model. The autoencoder can also be used to iteratively optimize the receiver and the transmitter. The receiver uses supervised learning, and the transmitter uses reinforcement learning based on the gradient of the loss function to optimize, which can be directly trained in the actual channel. Therefore, the transmission algorithm based on deep learning can obtain differentiable transmission layer parameters.

[0166] In a possible implementation manner, after obtaining the network state N, the control module 131 can establish a functional relationship between the experience index q and the differentiable rendering layer parameter R, the differentiable rendering layer parameter R, the differentiable coding layer parameter E and the differentiable transmission layer parameter T under the above network state parameters by means of a neural network, that is, q|N=f(R,E,T|N).

[0167] Then, the control module 131 obtains the optimal value of the above functional relationship, and determines the differentiable rendering layer parameter R0, the differentiable encoding layer parameter E0 and the differentiable transmission layer parameter T0 corresponding to the optimal value.

[0168] Finally, when the cloud rendering platform 130 performs rendering through the rendering module 101, the three-dimensional model data corresponding to the user request is rendered based on the micro-rendering layer parameter RO to generate the target video data. When encoding through the encoding module 102, the target video data is encoded based on the micro-coding layer parameter E0 to obtain the target video code stream. When transmitting through the transmission module, the target video code stream is sent to the terminal device via the communication connection based on the micro-transmission layer parameter T0.

[0169] See also Figure 7 , Figure 7 The flowchart of a video processing method provided by an embodiment of the present application is shown in FIG. 1. Optionally, the method can be applied to a video processing device, the video processing device is arranged in at least one data center, and a communication connection is established between at least one data center and a terminal device.

[0170] In one implementation, the video processing device may be arranged in a cloud rendering platform in the aforementioned video processing system, and specifically may be one or more rendering nodes in the cloud rendering platform.

[0171] like Figure 7 The cloud video processing method shown may include one or more steps from step S701 to step S705. It should be understood that the present application does not limit the execution sequence, execution time, and execution times of the above one or more steps.

[0172] Steps S701 to S703 are specifically as follows:

[0173] Step S701: Acquire three-dimensional model data and network status parameters.

[0174] The 3D model data includes the original data to be rendered, and the network status parameters include one or more of the following: bandwidth, packet loss rate, latency, and jitter, etc. For detailed description, please refer to Figure 3 , I will not go into details here.

[0175] The video processing device is a device with computing capabilities. For example, the video processing device may include a controller, a processor, a virtual computing instance, etc., where the virtual computing instance may be a virtual machine or a container. Alternatively, the video processing device may include a server, a host, etc. For a detailed description, please refer to Figure 2A , I will not go into details here.

[0176] Step S702: Determine rendering parameters according to network status parameters.

[0177] The rendering parameters include one or more of the following: frame rate, resolution, and rendering layer status, etc.

[0178] In one implementation, the rendering layer includes a rendering layer turned on and a rendering layer turned off. The video processing device determines that the network state is normal according to the network state parameters, indicating that the high-quality video data obtained by rendering can be sent to the terminal device through the communication connection corresponding to the network state, and the rendering quality can be improved, thereby performing one of the following operations: increasing the frame rate, increasing the resolution, and turning on the target rendering layer in at least one rendering layer according to the rendering layer state. It can be understood that the higher the frame rate, the higher the resolution, and the more rendering layers, the higher the quality of the video data, and the more data the video data contains. For example, in a layered rendering method that divides the rendering layers by rendering effects, if the color layer, the highlight layer, and the shadow layer are more important rendering layers, and the reflection layer and the refraction layer are less important rendering layers, then when the video processing device determines that the network state is normal according to the network state parameters, the states of the reflection layer and the refraction layer can be set to the rendering layer turned on to enhance the rendering effect.

[0179] In one implementation, the rendering layer state includes rendering layer on and rendering layer off. The video processing device determines that the network state is abnormal (for example, high packet loss rate, long delay, large network jitter, etc.) according to the network state parameters, indicating that the high-quality video data rendered according to the relatively strong rendering computing power cannot be transmitted to the device through the communication connection corresponding to the network state. Therefore, the rendering quality can be reduced, thereby performing one of the following operations: reducing the frame rate, reducing the resolution, and closing the target rendering layer in at least one rendering layer according to the rendering layer state. It can be understood that the lower the frame rate, the lower the resolution, and the fewer the number of rendering layers, the lower the quality of the video data, and the less data contained in the video data. Therefore, it can be guaranteed that the video data rendered with a lower rendering computing power can be transmitted to the terminal device through the communication connection corresponding to the network state. For example, in a layered rendering method in which rendering layers are divided by rendering effects, if the color layer, highlight layer, and shadow layer are more important rendering layers, and the reflection layer and refraction layer are less important rendering layers, then when the video processing device determines that the network status is abnormal based on network status parameters, the status of the reflection layer or the refraction layer can be set to render layer closed, so as to reduce the rendering effect while ensuring the transmission of basic content and save computing power.

[0180] As a possible implementation, the video processing device determines the target video processing strategy with the highest matching degree from multiple video processing strategies in the video processing strategy library according to the matching degree between the network status parameter and multiple preset network statuses, wherein the video processing strategy library stores multiple video processing strategies corresponding to multiple preset network statuses, the preset network status includes at least one network status parameter, and each of the multiple video processing strategies includes rendering parameters. Then, the video processing device determines the rendering parameters based on the target video processing strategy. For a detailed description, please refer to Figure 4, I will not go into details here.

[0181] As another possible implementation, the video processing device calculates the target information entropy contained in the content (i.e., the target video data) output by the renderer according to the network status parameters; and determines the rendering parameters based on the target information entropy. Figure 5 , I will not go into details here.

[0182] In one implementation, the video processing device calculates the information entropy interval contained in the code stream output by the encoder (i.e., the target video code stream) according to the network state parameters, wherein the code stream is obtained after the encoder encodes the content output by the renderer. Then, the video processing device determines the target information entropy contained in the rendered content (i.e., the target video data) output by the renderer according to the information entropy interval. For a detailed description, please refer to Figure 5 , I will not go into details here.

[0183] As another possible implementation, the video processing device determines the rendering parameters through a neural network according to the network state parameters. Figure 6 , I will not go into details here.

[0184] Step S703: Rendering the three-dimensional model data based on the rendering parameters to generate target video data.

[0185] The target video data includes at least one rendering layer.

[0186] Exemplarily, the video processing device may perform layered rendering on the original model data generated by the rendering application to obtain a rendering image including multiple rendering layers, in which case the rendering parameters include rendering parameters corresponding to different rendering layers. Therefore, the video processing device may render the three-dimensional model data based on the rendering parameters corresponding to the different rendering layers to generate target video data. As a possible implementation, the video processing device may render the three-dimensional model data based on the rendering parameters determined based on the target video processing strategy to generate target video data. For a detailed description, please refer to Figure 4 , I will not go into details here.

[0187] As a possible implementation, the video processing device may render the three-dimensional model data according to the rendering parameters determined based on the target information entropy to generate the target video data. Figure 5 , I will not go into details here.

[0188] As a possible implementation, the video processing device may render the three-dimensional model data according to the rendering parameters determined by the neural network to generate the target video data. Figure 6 , I will not go into details here.

[0189] Step S704: encode the target video data to obtain a target video code stream.

[0190] As a possible implementation, the target video processing strategy also includes encoding parameters. The video processing device can determine the encoding parameters based on the target video processing strategy, and then encode the target video data based on the encoding parameters to obtain the target video code stream. Figure 4 , I will not go into details here.

[0191] As a possible implementation, the video processing device determines the encoding parameters through a neural network according to the network state parameters, encodes the target video data based on the encoding parameters, and obtains the target video code stream. Figure 6 , I will not go into details here.

[0192] Step S705: Send the target video code stream to the terminal device via the communication connection.

[0193] As a possible implementation, the target video processing strategy also includes transmission parameters. The video processing device can determine the transmission parameters based on the target video processing strategy, and then send the target video code stream to the terminal device via the communication connection based on the transmission parameters. Figure 4 , I will not go into details here.

[0194] As a possible implementation, the video processing device determines the transmission parameters through a neural network according to the network status parameters, and then sends the target video code stream to the terminal device via a communication connection based on the transmission parameters. Figure 6 , I will not go into details here.

[0195] It is understandable that the device provided in the embodiment of the present application includes hardware structures, software units, or combinations of hardware structures and software structures, etc., for executing the functions in the above-mentioned method embodiments in order to realize the functions in the above-mentioned method embodiments. It should be easily appreciated by those skilled in the art that the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different device implementations to implement the aforementioned method embodiments in different usage scenarios, and different implementations of the device should not be considered to exceed the scope of the embodiments of the present application.

[0196] The embodiments of the present application may divide the device into functional units. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one functional unit. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0197] The method of the embodiment of the present application is described above, and the device of the embodiment of the present application is provided below.

[0198] See also Figure 8 , Figure 8 80 is a schematic diagram of the structure of a video processing device 80 provided in an embodiment of the present application. The video processing device 80 may include a communication module 801 and a processing module 802. The video processing device 80 is used to implement the aforementioned video processing method.

[0199] Optionally, the video processing device 80 may be arranged in at least one data center, for example, Figure 2A , Figure 2B , Figure 3-Figure 7 The video processing device 80 may be a stand-alone device or a device included in a stand-alone device, such as a chip, a software module, or an integrated circuit.

[0200] Wherein, the communication module 801 and the processing module 802 can be implemented by software or by hardware. Exemplarily, the implementation of the processing module 802 is described below by taking the processing module 802 as an example. Similarly, the implementation of the communication module 801 can refer to the implementation of the processing module 802.

[0201] As an example of a software functional unit, the processing module 802 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the processing module 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0202] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0203] As an example of a hardware functional unit, the processing module 802 may include at least one computing device, such as a server, etc. Alternatively, the processing module 802 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0204] The multiple computing devices included in the processing module 802 can be distributed in the same region or in different regions. The multiple computing devices included in the processing module 802 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing module 802 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0205] It should be noted that, in other embodiments, the processing module 802 can be used to execute any step in the video processing method, and the communication module 801 can be used to execute any step in the video processing method. The steps that the processing module 802 and the communication module 801 are responsible for implementing can be specified as needed, and the processing module 802 and the communication module 801 respectively implement different steps in the video processing method to achieve all the functions of the video processing device. Exemplarily, the communication module 801 can be used to execute the actions performed by the above-mentioned transmission module or network detection module, and the processing module can be used to execute the actions performed by the above-mentioned rendering module, encoding module, or control module.

[0206] In a possible implementation, the communication module is used to obtain the three-dimensional model data and network status parameters, where the network status parameters are parameters related to the network status of the communication connection, and the network status parameters include one or more of the following: bandwidth, packet loss rate, delay, and jitter;

[0207] The processing module is used to determine rendering parameters according to the network status parameters, wherein the rendering parameters include one or more of the following: frame rate, resolution, and rendering layer status;

[0208] The processing module is further used to render the three-dimensional model data based on the rendering parameters to generate target video data, wherein the target video data includes at least one rendering layer;

[0209] The processing module is further used to encode the target video data to obtain a target video code stream;

[0210] The communication module is further used to send the target video code stream to the terminal device via the communication connection.

[0211] In a possible implementation manner, the processing module is specifically configured to:

[0212] According to the matching degree between the network state parameter and a plurality of preset network states, determining the target video processing strategy with the highest matching degree from a plurality of video processing strategies in a video processing strategy library, wherein the video processing strategy library stores a plurality of video processing strategies corresponding to a plurality of preset network states, the preset network state includes at least one of the network state parameters, and each of the plurality of video processing strategies includes the rendering parameter;

[0213] The rendering parameters are determined based on the target video processing strategy.

[0214] In a possible implementation manner, the processing module is further used to determine the encoding parameter and the transmission parameter based on the target video processing strategy;

[0215] The processing module is specifically used to encode the target video data based on the encoding parameters to obtain a target video code stream;

[0216] The communication module is specifically used to: send the target video code stream to the terminal device via the communication connection based on the transmission parameters.

[0217] In a possible implementation, the device further includes a renderer, and the processing module is specifically configured to:

[0218] Calculate the target information entropy contained in the target video data according to the network state parameter;

[0219] The rendering parameters are determined based on the target information entropy.

[0220] In a possible implementation manner, the processing module is specifically configured to:

[0221] Calculate the information entropy interval contained in the target video bitstream according to the network state parameter;

[0222] The target information entropy included in the target video data is determined according to the information entropy interval.

[0223] In a possible implementation manner, the processing module is specifically configured to:

[0224] The rendering parameters are determined through a neural network according to the network state parameters.

[0225] In a possible implementation manner, the processing module is further configured to determine encoding parameters and transmission parameters through the neural network according to the network state parameters;

[0226] The processing module is specifically used to encode the target video data based on the encoding parameters to obtain a target video code stream;

[0227] The communication module is specifically configured to send the target video code stream to the terminal device via the communication connection based on the transmission parameters.

[0228] In a possible implementation manner, the rendering layer state includes rendering layer on and rendering layer off, and the processing module is specifically configured to:

[0229] Determine that the network status is normal according to the network status parameter, and perform one of the following operations:

[0230] The frame rate is increased, the resolution is increased, and a target rendering layer in the at least one rendering layer is enabled according to the rendering layer state.

[0231] In a possible implementation manner, the rendering layer state includes rendering layer on and rendering layer off, and the processing module is specifically configured to:

[0232] Determine that the network status is abnormal according to the network status parameter, and perform one of the following operations:

[0233] The frame rate is reduced, the resolution is reduced, and a target rendering layer in the at least one rendering layer is closed according to the rendering layer state.

[0234] Fig. 9The figure shows a schematic diagram of the structure of a computing device 90 provided in an embodiment of the present application. The computing device 90 is a device with computing capabilities, where the device can be a physical device, such as a controller, a processor, a server (such as a rack server), a host, etc., or a virtual device, such as a virtual machine, a container, etc.

[0235] like Fig. 9 As shown, the computing device 90 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate through the bus 902. The computing device 90 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 90.

[0236] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 902 may include a path for transmitting information between various components of the computing device 90 (eg, the memory 906, the processor 904, and the communication interface 908).

[0237] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0238] The memory 906 may include a volatile memory, such as a random access memory (RAM). The processor 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0239] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned communication module and processing module, thereby implementing the aforementioned video processing method. That is, the memory 906 stores instructions for executing the video processing method.

[0240] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 90 and other devices or a communication network.

[0241] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0242] like Fig.10 The diagram is a schematic diagram of a computing device cluster provided in an embodiment of the present application, wherein the computing device cluster includes at least one computing device 90. The memory 906 in one or more computing devices 90 in the computing device cluster may store the same instructions for executing the video processing method.

[0243] In some possible implementations, the memory 906 of one or more computing devices 90 in the computing device cluster may also store partial instructions for executing the video processing method. In other words, the combination of one or more computing devices 90 may jointly execute instructions for executing the video processing method.

[0244] It should be noted that the memory 906 in different computing devices 90 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the video processing device. That is, the instructions stored in the memory 906 in different computing devices 90 can implement the functions of one or more modules in the communication module and the processing module.

[0245] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11The diagram shows a schematic diagram of one or more computing devices in a computing device cluster provided by an embodiment of the present application being connected through a network, and two computing devices 90A and 90B are connected through a network. Specifically, the network is connected through a communication interface in each computing device. In this type of possible implementation, the memory 906 in the computing device 90A stores instructions for executing the functions of the processing module. At the same time, the memory 906 in the computing device 90B stores instructions for executing the functions of the communication module.

[0246] Fig.11 The connection method between the computing device clusters shown may be based on the consideration that the video processing method provided in the present application requires, for example, a large amount of data storage, and therefore it is considered to entrust the function of the communication module to the computing device 90B for execution.

[0247] It should be understood that Fig.11 The functions of the computing device 90A shown in FIG. 1 may also be completed by multiple computing devices 90. Similarly, the functions of the computing device 90B may also be completed by multiple computing devices 90.

[0248] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.10 and Fig.11 The connection mode of the computing device cluster is different in that the memory 906 in one or more computing devices 90 in the computing device cluster may store the same instructions for executing the video processing method.

[0249] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the video processing method.

[0250] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the video processing method.

[0251] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method, characterized in that: Applied to a video processing device, the video processing device is arranged in at least one data center, a communication connection is established between the at least one data center and a terminal device, the method comprises: Acquire three-dimensional model data and network status parameters, wherein the network status parameters are parameters related to the network status of the communication connection, and the network status parameters include one or more of the following: bandwidth, packet loss rate, delay and jitter; Determine rendering parameters according to the network status parameters, wherein the rendering parameters include one or more of the following: frame rate, resolution, and rendering layer status; Rendering the three-dimensional model data based on the rendering parameters to generate target video data, wherein the target video data includes at least one rendering layer; Encoding the target video data to obtain a target video code stream; The target video code stream is sent to the terminal device via the communication connection.

2. The method according to claim 1, characterized in that The determining of rendering parameters according to the network status parameters includes: According to the matching degree between the network state parameter and a plurality of preset network states, determining the target video processing strategy with the highest matching degree from a plurality of video processing strategies in a video processing strategy library, wherein the video processing strategy library stores a plurality of video processing strategies corresponding to a plurality of preset network states, the preset network state includes at least one of the network state parameters, and each of the plurality of video processing strategies includes the rendering parameter; The rendering parameters are determined based on the target video processing strategy.

3. The method according to claim 1 or 2, characterized in that: The target video processing strategy also includes encoding parameters and transmission parameters, and the method also includes: Determining the encoding parameters and the transmission parameters based on the target video processing strategy; The encoding of the target video data to obtain a target video code stream, and sending the target video code stream to the terminal device via the communication connection, comprises: Encode the target video data based on the encoding parameters to obtain a target video code stream; The target video code stream is sent to the terminal device via the communication connection based on the transmission parameters.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of rendering parameters according to the network status parameters includes: Calculate the target information entropy contained in the target video data according to the network state parameter; The rendering parameters are determined based on the target information entropy.

5. The method according to claim 4, characterized in that The step of calculating the target information entropy contained in the target video data according to the network state parameter includes: Calculate the information entropy interval contained in the target video bitstream according to the network state parameter; The target information entropy included in the target video data is determined according to the information entropy interval.

6. The method according to any one of claims 1 to 5, characterized in that: The determining of rendering parameters according to the network status parameters includes: The rendering parameters are determined through a neural network according to the network state parameters.

7. The method according to claim 6, characterized in that The method further comprises: Determining encoding parameters and transmission parameters through the neural network according to the network state parameters; The encoding of the target video data to obtain a target video code stream, and sending the target video code stream to the terminal device via the communication connection, comprises: Encode the target video data based on the encoding parameters to obtain a target video code stream; The target video code stream is sent to the terminal device via the communication connection based on the transmission parameters.

8. The method according to any one of claims 1 to 7, characterized in that: The rendering layer state includes rendering layer on and rendering layer off, and determining the rendering parameter according to the network state parameter includes: Determine that the network status is normal according to the network status parameter, and perform one of the following operations: The frame rate is increased, the resolution is increased, and a target rendering layer in the at least one rendering layer is enabled according to the rendering layer state.

9. The method according to claims 1 to 7, characterized in that The rendering layer state includes rendering layer on and rendering layer off, and determining rendering parameters according to the network state parameters includes: Determine that the network status is abnormal according to the network status parameter, and perform one of the following operations: The frame rate is reduced, the resolution is reduced, and a target rendering layer in the at least one rendering layer is closed according to the rendering layer state.

10. A video processing device, characterized in that: The video processing device is arranged in at least one data center, a communication connection is established between the at least one data center and the terminal device, and the video processing device comprises a communication module and a processing module, wherein: The communication module is used to obtain the three-dimensional model data and network status parameters, where the network status parameters are parameters related to the network status of the communication connection, and the network status parameters include one or more of the following: bandwidth, packet loss rate, delay and jitter; The processing module is used to determine rendering parameters according to the network status parameters, wherein the rendering parameters include one or more of the following: frame rate, resolution, and rendering layer status; The processing module is further used to render the three-dimensional model data based on the rendering parameters to generate target video data, wherein the target video data includes at least one rendering layer; The processing module is further used to encode the target video data to obtain a target video code stream; The communication module is further used to send the target video code stream to the terminal device via the communication connection.

11. The device according to claim 10, characterized in that The processing module is specifically used for: According to the matching degree between the network state parameter and a plurality of preset network states, determining the target video processing strategy with the highest matching degree from a plurality of video processing strategies in a video processing strategy library, wherein the video processing strategy library stores a plurality of video processing strategies corresponding to a plurality of preset network states, the preset network state includes at least one of the network state parameters, and each of the plurality of video processing strategies includes the rendering parameter; The rendering parameters are determined based on the target video processing strategy.

12. The device according to claim 10 or 11, characterized in that The processing module is further used to determine the encoding parameter and the transmission parameter based on the target video processing strategy; The processing module is specifically used to encode the target video data based on the encoding parameters to obtain a target video code stream; The communication module is specifically used to: send the target video code stream to the terminal device via the communication connection based on the transmission parameters.

13. The device according to any one of claims 10 to 12, characterized in that The processing module is specifically used for: Calculate the target information entropy contained in the target video data according to the network state parameter; The rendering parameters are determined based on the target information entropy.

14. The device according to claim 13, characterized in that The processing module is specifically used for: Calculate the information entropy interval contained in the target video bitstream according to the network state parameter; The target information entropy included in the target video data is determined according to the information entropy interval.

15. The device according to any one of claims 10 to 14, characterized in that The processing module is specifically used for: The rendering parameters are determined through a neural network according to the network state parameters.

16. The device according to claim 15, characterized in that The processing module is further used to determine encoding parameters and transmission parameters through the neural network according to the network state parameters; The processing module is specifically used to encode the target video data based on the encoding parameters to obtain a target video code stream; The communication module is specifically configured to send the target video code stream to the terminal device via the communication connection based on the transmission parameters.

17. The device according to any one of claims 10 to 16, characterized in that The rendering layer state includes rendering layer on and rendering layer off, and the processing module is specifically used for: Determine that the network status is normal according to the network status parameter, and perform one of the following operations: The frame rate is increased, the resolution is increased, and a target rendering layer in the at least one rendering layer is enabled according to the rendering layer state.

18. The device according to any one of claims 10 to 16, characterized in that The rendering layer state includes rendering layer on and rendering layer off, and the processing module is specifically used for: Determine that the network status is abnormal according to the network status parameter, and perform one of the following operations: The frame rate is reduced, the resolution is reduced, and a target rendering layer in the at least one rendering layer is closed according to the rendering layer state.

19. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

21. A computer program product comprising instructions, characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

Images