Coding method, apparatus and device, decoding method, apparatus and device, storage medium and program product

Abstract

The present application belongs to the technical field of coding and decoding. Disclosed are a coding method, apparatus and device, a decoding method, apparatus and device, a storage medium and a program product. The coding method comprises: acquiring scene data of the current frame in a three-dimensional virtual scene; coding the scene data into a code stream; and, on the basis of the scene data, coding a rendered image of the current frame into the code stream. It can be seen that the scene data is transmitted by means of the code stream, such that a decoding end (or a client receiving the code stream) does not need to download and store in advance the scene data required for running the three-dimensional virtual scene, thus reducing the requirement for running the three-dimensional virtual scene at an end side. When the scene data of the current frame is transmitted, the scene data of a coding end and that of the decoding end can be kept consistent. On this basis, during the process of coding the rendered image of the current frame, the coding end can use the scene data to optimize the coding process of the rendered image, so as to reduce the code rate and computing overheads of coding the rendered image and the bandwidth resource for transmitting the code stream between the coding end and the decoding end.

Description

Coding and decoding method, device, equipment, storage medium and program product

[0001] This application claims priority to Chinese patent application No. 202410205634.4, filed on February 23, 2024, entitled “Encoding and decoding method, device, equipment, storage medium and program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of coding and decoding technology, and in particular to a coding and decoding method, apparatus, device, storage medium and program product. Background Art

[0003] With advances in technology and the development of cloud computing, cloud rendering has become a vital tool in industries such as gaming, design, architecture, and film and television. Cloud rendering utilizes a cloud platform to generate rendered images, encode these images into a bitstream, and transmit this bitstream to the client. This not only reduces client hardware and maintenance costs, but also leverages the powerful performance and elastic scalability of the cloud platform to improve rendering efficiency and quality.

[0004] Taking the gaming industry as an example, cloud gaming is an online gaming service based on cloud computing. Game logic and game assets are stored on cloud servers, allowing players to remotely control the game over the network without having to download and install the game on the local client. In cloud gaming mode, the cloud platform receives player operation information transmitted by the client, renders the game scene for the current frame, and generates a rendered image of the current frame. This rendered image is then encoded and compressed, and then transmitted over the network to the client for display.

[0005] However, cloud rendering requires the cloud platform to encode and transmit the entire rendered image, resulting in a high bitrate bandwidth when transmitting the rendered image. Summary of the Invention

[0006] This application provides a coding and decoding method, apparatus, device, storage medium, and program product that can reduce the bitstream bandwidth when transmitting rendered images between a cloud platform and a client. The technical solution is as follows:

[0007] In a first aspect, a coding method is provided, comprising: obtaining scene data of a current frame in a three-dimensional virtual scene; encoding the scene data into a bitstream; and encoding a rendered image of the current frame into the bitstream based on the scene data.

[0008] In the present application, scene data is transmitted via a code stream, so that the decoding end (or the client receiving the code stream) does not need to pre-download and store the scene data required to run the three-dimensional virtual scene. At the same time, when transmitting the scene data of the current frame, the scene data of the encoding end and the decoding end can remain consistent. Based on this, the encoding end can effectively utilize the scene data in the process of encoding the rendered image of the current frame, and optimize the encoding process of the rendered image to reduce the bit rate and computational overhead of the encoded rendered image. Furthermore, when using scene data to encode the rendered image can reduce the bit rate, this encoding method can also reduce the bandwidth resources for transmitting the code stream between the encoding end and the decoding end.

[0009] It should be noted that when the scene data at the encoding end and the decoding end remain consistent, if the encoding end uses the scene data to optimize the encoding process of the rendered image, the decoding end can decode the bitstream based on the same scene data and process the decoding information in a corresponding manner to obtain the rendered image reconstructed for the current frame.

[0010] In addition, for the game scene, the encoding end of the encoding method provided by the present application can be a cloud platform or a target server in the cloud platform, and the corresponding decoding end can be a client, that is, the cloud platform and the client use an end-cloud collaborative approach to run the three-dimensional virtual scene. After determining the scene data of the current frame, the cloud platform can encode the scene data into a code stream and send it to the client. In this way, the client does not need to pre-download and store the scene data required to run the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience. When the cloud platform and the client have the same scene data, the cloud platform can effectively use the scene data in the process of encoding the rendered image of the current frame and optimize the encoding process of the rendered image; correspondingly, the client can decode the code stream based on the same scene data and process the decoding information in a corresponding manner to obtain the rendered image reconstructed by the current frame. In this way, not only can the bit rate and computational overhead of the cloud platform's encoded rendered image be reduced, but the client's own processing power can also be effectively utilized, thereby reducing the bandwidth resources for transmitting the code stream between the cloud platform and the client.

[0011] In one possible implementation, encoding the rendered image of the current frame into the code stream based on the scene data includes: obtaining an intermediate rendering result of the current frame, where the intermediate rendering result is obtained by rendering the scene data; and encoding the rendered image based on the intermediate rendering result.

[0012] The intermediate rendering results are the prior information generated during the rendering of the current frame, which may include one or more of the computer graphic motion vector (CGMV) generated by the computer graphics method of the current frame, low-quality rendering results, position map, normal map, albedo map, specular intensity map, mesh identifier (Mesh ID), material identifier (Material ID), and depth map. These intermediate rendering results can be applied to the encoding process of the rendered image to optimize the encoding process.

[0013] Among them, the methods for encoding the rendered image of the current frame include intra-frame prediction and inter-frame prediction. Taking inter-frame prediction as an example, the process of encoding the current frame includes steps such as blocking, motion estimation, MV prediction, motion compensation, quantization, and entropy coding. Therefore, when the present application uses scene data to assist in encoding the rendered image of the current frame, it can optimize at least one of the above steps based on the intermediate rendering result of the current frame to achieve optimization of the encoding process of the rendered image of the current frame.

[0014] Based on the intermediate rendering results of the current frame, at least one of the steps in the encoding process, namely, dividing the image blocks, determining the reference frame for the current frame, determining the matching block for the current block, and determining the predicted MV for the current block, can be optimized. Optimizing each step yields a new encoding scheme. Combining the optimization methods for individual steps, this application provides four corresponding encoding schemes. These four encoding schemes are described in detail below.

[0015] In a first encoding scheme, encoding the rendered image based on the intermediate rendering result includes:

[0016] Based on the intermediate rendering result, a virtual reference image is generated; based on the virtual reference image, the rendered image is encoded into a bitstream.

[0017] The virtual reference image can be used as supplementary information for inter-frame coding to further remove video temporal redundancy.

[0018] In one possible implementation, the rendered image of the historical frame is reprojected based on the rendered image of the historical frame and the CGMV of the current frame to generate a virtual reference image corresponding to the current frame. Reprojection refers to the calculation process of finding the corresponding pixel point in the historical frame based on the CGMV for a pixel in the current frame.

[0019] When a virtual reference image is generated based on the intermediate rendering result, the rendered image of the current frame can be directly encoded using the virtual reference image. Alternatively, the virtual reference image can be used to determine a predicted image for encoding the current frame, and the predicted image can be used to encode the rendered image of the current frame. This application does not impose any restrictions on this, and you can flexibly choose based on actual needs and business scenarios.

[0020] It can be seen that the present application generates a virtual reference image based on the intermediate rendering result of the current frame. Since the similarity between the virtual reference image and the rendered image of the current frame is higher, the virtual reference image can provide a more accurate reference for encoding the rendered image of the current frame, which can improve the inter-frame prediction accuracy, thereby improving the coding efficiency and reducing the bit rate when encoding the rendered image of the current frame.

[0021] In a second encoding scheme, the rendering intermediate result of the current frame includes a depth map, and encoding the rendered image based on the rendering intermediate result includes:

[0022] Based on the depth map, partition information of a first image block is generated, where the first image block is the image block of the rendered image, and the partition information is used to divide the first image block to obtain a plurality of partitions; and the partition information is encoded into a bitstream.

[0023] Taking the geometric partitioning mode (GPM) as an example, the first image block may be any coding unit (CU) in the current frame rendering image, and the partitioning information of the first image block may be the position of a dividing line that divides the first image block into two.

[0024] In a possible implementation, generating segmentation information of the first image block based on the depth map includes:

[0025] Based on the depth map, a second division information set is determined from a first division information set, where the first division information set includes multiple division information, and the second division information set is a subset of the first division set; and the division information of the first image block is determined from the second division set.

[0026] The first division information set includes 64 division modes defined by GPM, that is, 64 dividing lines; the second division information set includes at least one of the 64 division modes.

[0027] In a possible implementation, determining the second segmentation information set from the first segmentation information set based on the depth map includes:

[0028] Based on the depth map, a position of a depth change in the first image block is determined; based on the position, the second partition information set is determined from the first partition information set.

[0029] The position where the depth changes in the first image block may be a contour line between the foreground object and the background area in the first image block.

[0030] In one possible implementation, the contour line in the first image block is analyzed to determine its direction and angle. According to the angle of the contour line, several division modes close to the position of the contour line are selected from the 64 division modes specified by GPM. and ρ, these screened The combination of and ρ is the second partition information set.

[0031] It can be seen that when using GPM to divide the image block in the current frame into smaller areas, the depth map of the current frame can be used to narrow the search range of the segmentation line, so that only those areas that are consistent with the filtered areas are considered when determining the segmentation pattern. The partitioning pattern corresponding to ρ. In this way, the 64 partitioning patterns that originally need to be traversed can be reduced to a smaller set, thereby significantly reducing the amount of GPM calculations, reducing computational complexity, and improving coding efficiency.

[0032] Based on the above-mentioned first encoding scheme and second encoding scheme, when encoding the rendered image in units of image blocks, if the rendering intermediate result includes the graphics motion vector (CGMV) of the current frame; then encoding the rendered image based on the rendering intermediate result includes: determining the motion vector (motion vector, MV) of the second image block based on the graphics motion vector, and encoding the MV of the second image block; the second image block is any image block in the rendered image.

[0033] The second image block may be an image block obtained by dividing in a traditional manner; or an image block obtained by dividing using the second encoding scheme described above. This application does not impose any restrictions on the division method of the second image block.

[0034] It should be noted that the MV of the second image block here is a real MV, which is different from the predicted MV below. That is, for the second image block, there is a real MV and a predicted MV. The predicted MV can be used in the encoding process of the real MV to reduce the amount of encoded data.

[0035] The third encoding scheme described below illustrates the implementation process of determining the MV of the second image block, and the fourth encoding scheme described below illustrates the implementation process of encoding the MV of the second image block. The third and fourth encoding schemes can be independently applied to the MV encoding process of the second image block, or can be used in combination to encode the MV of the second image block, and this application does not impose any restrictions on this.

[0036] In a third encoding scheme, the rendering intermediate result of the current frame includes a graphics motion vector (CGMV) of the current frame, and determining the MV of the second image block based on the graphics motion vector includes:

[0037] Based on the graphic motion vector, a predicted MV of a key pixel point in the second image block is determined, where the key pixel point includes one or more pixel points; based on the predicted MV of the key pixel point, a search range is determined; according to the search range, a matching block corresponding to the second image block is determined from a reference rendered image, where the reference rendered image is a rendered image of a reference frame corresponding to the current frame; and based on the second image block and the matching block corresponding to the second image block, the MV of the second image block is determined.

[0038] In a possible implementation, a motion trend of the second image block within a search window is determined according to the predicted MV of the key pixel point; and a search range is determined within the search window based on the motion trend.

[0039] The search window is an area in the rendered image of the reference frame (i.e., the reference rendered image) where a matching block of the second image block is searched. The search window includes multiple image blocks, and the image block located in the center area of ​​the search window corresponds to the position area of ​​the second image block in the rendered image of the current frame.

[0040] It should be noted that the reference rendered image can be a reference image determined by conventional means, i.e., an original reference image, or a virtual reference image determined by the first encoding scheme described above. Furthermore, the search range determined by the predicted MV of the key pixel is smaller than the search window size.

[0041] As can be seen, this application estimates the motion trend of the second image block based on the predicted MV of the key pixels in the second image block, thereby narrowing the search range for matching blocks and focusing them more closely on the possible best matching area. Because the search range is narrowed, only a small number of pixels need to be calculated for the rate-distortion cost when searching for matching blocks within the search range, greatly reducing the amount of data calculation required for the rate-distortion optimization (RDO) operation and lowering the computational complexity.

[0042] In a fourth encoding scheme, the rendering intermediate result of the current frame includes a graphics motion vector (CGMV) of the current frame, and encoding the MV of the second image block based on the graphics motion vector includes:

[0043] Based on the graphic motion vector, the predicted MV of the key pixel point in the second image block is determined, where the key pixel point includes one or more pixel points; based on the predicted MV of the key pixel point, the predicted MV of the second image block is determined; based on the predicted MV, the MV of the second image block is encoded.

[0044] In a possible implementation, determining the predicted motion vector of the second image block based on the predicted MV of the key pixel includes:

[0045] Constructing a candidate MV list of the second image block, wherein the candidate MV list includes the predicted MVs of the key pixels; and determining the predicted MV of the second image block based on the candidate MV list.

[0046] The candidate MV list of the second image block may be a candidate MV list of advanced motion vector prediction (AMVP) or a candidate MV list of Merge mode, and this application does not impose any limitation on this.

[0047] After determining the predicted MV for the key pixel in the second image block, the predicted MV for the key pixel is also added to the candidate MV list for the second image block. That is, the predicted MV for the key pixel participates in the calculation process of determining the predicted MV for the second image block. For each candidate MV in the candidate MV list, a similarity metric (e.g., rate-distortion cost) is calculated between each candidate MV and the (actual) MV of the second image block. The candidate MV that best matches the motion of the second image block is selected from the candidate MV list and determined as the predicted MV for the second image block.

[0048] Taking H.265 as an example, the methods of encoding the MV of the image block include AMVP and Merge mode. The following introduces the implementation process of encoding the second image block in these two ways.

[0049] In one possible implementation, encoding the MV of the second image block based on the predicted MV includes: obtaining an MV residual based on the MV of the second image block and the predicted MV of the second image block; encoding the MV residual into a bitstream; and encoding index information into the bitstream, the index information indicating a position of the predicted MV of the second image block in the candidate MV list.

[0050] In a possible implementation, encoding the MV of the second image block based on the predicted MV includes: encoding index information into the bitstream, where the index information indicates a position of the predicted MV of the second image block in the candidate MV list.

[0051] That is, when encoding the MV of the second image block using AMVP, the residual between the predicted MV of the second image block and the (actual) MV of the second image block, as well as the index information of the predicted MV in the candidate MV list, are encoded into the bitstream. When encoding the MV of the second image block in Merge mode, only the index information of the predicted MV in the candidate MV list is encoded into the bitstream. It can be seen that the information encoded into the bitstream by the two methods is different. Compared with AMVP, the amount of data used to encode the MV of the second image block in Merge mode is smaller, further reducing the encoding bitrate.

[0052] It can be seen that when encoding the MV of the second image, the present application adds the predicted MV of the key pixel points in the second image block to the candidate MV list, thereby improving the similarity between the predicted MV and the MV of the second image block, thereby reducing the encoding bit rate of the MV of the second image block encoded in AMVP and Merge modes, and improving the encoding efficiency.

[0053] In one possible implementation, the scene data includes local scene data and / or scene change data, where the local scene data refers to scene data newly added to the current frame relative to the historical frame, and the scene change data refers to scene data whose state has changed relative to the historical frame.

[0054] That is, the scene data of the current frame may include only the local scene data of the current frame, or only the scene change data of the current frame, or may include both the local scene data and the scene change data of the current frame. This application does not impose any restrictions on this.

[0055] In one possible implementation, the method is applied to a target server, and obtaining scene data of a current frame in a three-dimensional virtual scene includes: receiving scene interaction information sent by a client, the scene interaction information indicating an operation triggered by an operator in the three-dimensional virtual scene, and a position of the operator in the three-dimensional virtual scene; and determining the scene data of the current frame based on the scene interaction information.

[0056] Among them, the target server can be a server that performs image encoding tasks in the cloud platform, such as the server with the shortest communication distance between the cloud platform and the client; the target server can also be a server located close to the client on the communication connection between the cloud platform and the client, such as the client's proximal server or local server.

[0057] When the communication distance between the target server and the client is the shortest, the communication delay between the target server and the client is the shortest, the transmission speed of the code stream is faster, and the communication performance is better.

[0058] In one possible implementation, determining the scene data of the current frame based on the scene interaction information includes: determining the scene data within a target range as the scene data of the current frame based on the scene interaction information, where the target range refers to a range with a target size centered on the position of the operator in the three-dimensional virtual scene and extending to the surrounding area; or determining the posture of a virtual camera in the three-dimensional virtual scene based on the scene interaction information, and determining the scene data within a visible range of the virtual camera as the scene data of the current frame based on the posture of the virtual camera.

[0059] For the current frame, this application does not restrict the coverage of the scene data (or the size of the scene data), and can be flexibly selected based on the network bandwidth and latency requirements between the client and the target server. Once the target server and the client have the same scene data, it can be guaranteed that the target server and the client can render the same intermediate rendering results to assist in encoding and decoding the rendered image of the current frame.

[0060] Since the encoding and decoding method of the present application requires that both the target server and the client have the scene data of the current frame and have image rendering capabilities, they can render the scene data to obtain a rendering intermediate result, and assist in encoding and decoding the rendered image of the current frame based on the same rendering intermediate result. Therefore, whether the target server and the client run the end-cloud collaboration mode, that is, whether the target server uses the encoding method provided by the present application to generate the bit stream, needs to be determined in combination with the client's image rendering capabilities, and / or the network bandwidth between the target server and the client.

[0061] In one possible implementation, the method is applied to a target server, and the method further includes: obtaining device information of the client, where the device information indicates the image rendering capability of the client; and determining a running end-cloud collaboration mode based on the device information of the client.

[0062] Among them, the device information includes multiple device parameters of the device used by the client, such as device type, memory and running memory size, graphics processing unit (GPU) model, graphics driver version number, operating system type, etc. This application does not impose any restrictions on this.

[0063] As an example, the present application may score various device parameters of the client based on the device information of the client and the thresholds corresponding to the various device parameters, and perform weighted summation on the scores of the various device parameters to determine the rendering capability level of the client.

[0064] Among them, the rendering capability level includes the first rendering level, the second rendering level and the third rendering level. The rendering capability level is the first rendering level, which means that the client has strong image rendering capability and can independently render the scene data to obtain the rendered image and rendering intermediate results of the current frame. Its image rendering capability is close to that of the cloud platform (or target server); the rendering capability level is the second rendering level, which means that the client has strong image rendering capability but not as good as the cloud platform. The client can render the scene data to obtain the rendering intermediate results of the current frame, but cannot render a rendered image with an effect close to that of the cloud platform; the rendering capability level is the third rendering level, which means that the client has weak image rendering capability or does not have image rendering capability and can only parse the code stream but cannot render the scene data of the current frame to obtain the rendering intermediate results.

[0065] Based on this, if it is determined that the rendering capability level of the client is the second rendering level, the end-cloud collaboration mode is run. Of course, when the rendering capability level of the client is the first rendering level, the end-cloud collaboration mode can also be run.

[0066] It can be seen from this that the image rendering capability of the client is evaluated in combination with the device information of the client, and the technical solution of this application is executed when it is determined that the client supports the end-cloud collaborative mode, so as to avoid the situation where the rendered image of the current frame cannot be reconstructed based on the code stream sent by the cloud platform due to insufficient image rendering capability of the client.

[0067] In one possible implementation, when it is determined that the client's image rendering capability is insufficient, for example, the client's rendering capability level is the third rendering level, the target server can send the rendered image to the client using the traditional video streaming method, that is, the target server encodes the rendered image of the current frame into a bitstream based on the rendered image of the historical frame, and sends the bitstream to the client.

[0068] In a possible implementation, the method is applied to a target server, and the method further includes: obtaining a network bandwidth between the target server and a client; and determining an end-cloud collaboration mode based on the network bandwidth.

[0069] It should be noted that in the end-cloud collaboration mode, the target server and client need to transmit the rendered image of the current frame, which is coded with scene data, as well as the bitstream containing the scene data of the current frame. This places certain requirements on the network bandwidth between the target server and the client. Therefore, during communication with the client, the target server can detect the network bandwidth between the target server and the client in real time and determine whether the current network bandwidth meets the operating conditions of the end-cloud collaboration mode.

[0070] As an example, if the network bandwidth between the target server and the client is greater than or equal to the bandwidth threshold, it is determined that the network bandwidth meets the operating conditions of the end-cloud collaboration mode, and the end-cloud collaboration mode can be run between the target server and the client, that is, the technical solution provided in this application is used to encode and decode the rendered image of the current frame.

[0071] If the network bandwidth between the target server and the client does not meet the operating conditions of the end-cloud collaboration mode, and the scene data of the current frame is stored in the cloud platform, the target server in the cloud platform uses the historical frame image to encode the rendered image of the current frame, and sends the code stream encoded with the rendered image of the current frame to the client.

[0072] In a second aspect, a decoding method is provided, the method comprising:

[0073] The scene data of the current frame in the three-dimensional virtual scene is obtained by decoding the code stream; and the code stream is decoded based on the scene data to obtain a rendered image reconstructed from the current frame.

[0074] As can be seen from this, the cloud platform and client in this application use a terminal-cloud collaborative mode to run a three-dimensional virtual scene. After the cloud platform sends the scene data of the current frame and the rendered image of the current frame encoded using the scene data to the client via a code stream, the client, in combination with its own image rendering capabilities, uses the scene data of the current frame for auxiliary decoding to obtain a rendered image reconstructed from the current frame. In this process, the client does not need to pre-download and store the scene data required to run the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience. Moreover, when the scene data of the cloud platform and the client can be consistent, the cloud platform can effectively utilize the scene data during the encoding process of the rendered image of the current frame and optimize the encoding process of the rendered image to reduce the bit rate and computational overhead of the encoded rendered image. Furthermore, when using scene data to encode the rendered image can reduce the bit rate, this encoding method can also reduce the bandwidth resources required to transmit the code stream between the encoding and decoding ends. Correspondingly, after parsing and obtaining the scene data of the current frame, the client can decode the code stream based on the same scene data and process the decoding information in a corresponding manner to obtain a rendered image reconstructed from the current frame. This not only reduces the bit rate and computational overhead of encoding and rendering images on the cloud platform, but also effectively utilizes the client's own processing power, thereby reducing the bandwidth resources for transmitting code streams between the cloud platform and the client.

[0075] In a possible implementation, decoding the code stream based on the scene data includes:

[0076] Obtaining an intermediate rendering result of the current frame, where the intermediate rendering result is obtained by rendering the scene data; decoding the code stream based on the intermediate rendering result to obtain a rendered image reconstructed from the current frame.

[0077] The intermediate rendering results are a priori information generated during the rendering of the current frame. They may include one or more of the following: CGMV generated by computer graphics methods for the current frame, low-quality rendering results, position maps, normal maps, albedo maps, specular intensity maps, bin identifiers, material identifiers, and depth maps. These intermediate rendering results can be used in the decoding process of the rendered image to optimize the decoding process.

[0078] It can be seen that in the process of decoding and obtaining the rendered image reconstructed by the current frame, the client also participates in completing part of the image rendering task to obtain the intermediate rendering result of the current frame, effectively utilizing the client's own image rendering capabilities.

[0079] When the information carried in the bitstream is different, the implementation process of decoding the bitstream based on the intermediate rendering result of the current frame to obtain the rendered image reconstructed by the current frame will also be different. Next, for the encoding scheme shown in the first aspect above, this application provides two corresponding decoding schemes, which are introduced below.

[0080] In a first decoding solution, decoding the code stream based on the rendering intermediate result includes:

[0081] Based on the intermediate rendering result, a virtual reference image is generated; and based on the virtual reference image, the code stream is decoded to obtain a rendered image reconstructed from the current frame.

[0082] The virtual reference image can be used as supplementary information for inter-frame coding to further remove video temporal redundancy.

[0083] The first decoding scheme corresponds to the first encoding scheme shown in the first aspect. After obtaining the intermediate rendering result of the current frame, the intermediate rendering result can be used to generate a virtual reference image, and the virtual reference image can be used to reconstruct the rendered image of the current frame.

[0084] In a second decoding scheme, the rendering intermediate result includes a graphics motion vector (CGMV) of the current frame, and decoding the code stream based on the rendering intermediate result includes:

[0085] The code stream is decoded based on the graphic motion vector to obtain a motion vector MV of a second image block, where the second image block is any image block in the rendered image.

[0086] Regarding the second image block, as described in the first aspect above, when encoding the MV of the second image block, the Merge mode can be used to encode index information indicating the position of the predicted MV in the candidate MV list into the bitstream, or the AMVP method can be used to encode the index information and the MV residual corresponding to the second image block into the bitstream. Correspondingly, decoding the bitstream based on the graphic motion vector to obtain the motion vector MV of the second image block also includes two implementation methods.

[0087] In a first implementation, decoding the code stream based on the graphic motion vector includes:

[0088] Based on the graphic motion vector, determine the predicted MV of the key pixel point in the second image block, where the key pixel point includes one or more pixel points; construct a candidate MV list of the second image block, where the candidate MV list includes the predicted MV of the key pixel point; obtain the MV of the second image block from the candidate MV list based on index information, where the index information is obtained from the code stream, and the index information indicates the position of the MV of the second image block in the candidate motion vector list.

[0089] In a second implementation, decoding the code stream based on the graphic motion vector includes:

[0090] Based on the graphic motion vector, determine the predicted MV of the key pixel point in the second image block, where the key pixel point includes one or more pixel points; construct a candidate MV list of the second image block, where the candidate MV list includes the predicted MV of the key pixel point; obtain the candidate MV of the second image block from the candidate MV list based on index information, where the index information is obtained from the code stream, and the index information indicates the position of the MV of the second image block in the candidate MV list; obtain the MV of the second image block based on the candidate MV of the second image block and the MV residual corresponding to the second image block, where the MV residual is obtained from the code stream.

[0091] That is, regardless of AMVP or Merge mode, after determining the predicted MV of the key pixel point in the second image block, the present application also adds the predicted MV of the key pixel point to the candidate MV list of the second image block, that is, the predicted MV of the key pixel point will participate in the calculation process of determining the predicted MV of the second image block.

[0092] The difference between AMVP and Merge modes is that after determining the candidate MV for the second image block from the candidate MV list (equivalent to the "predicted MV of the second image block" described on the encoding side), AMVP determines the MV of the second image block based on the MV residual parsed in the bitstream and the candidate MV. Merge mode, on the other hand, directly determines the candidate MV as the MV of the second image block.

[0093] The above-mentioned second decoding scheme corresponds to the fourth encoding scheme shown in the first aspect. After obtaining the intermediate rendering result of the current frame, the intermediate rendering result can be used to determine the predicted MV of the key pixel points of the second image block in the current frame, and then the predicted MV of the key pixel points of the second image block can be used to determine the MV of the second image block, and then the MV can be used to find the matching block to reconstruct the rendered image of the current frame.

[0094] In one possible implementation, the scene data includes local scene data and / or scene change data, where the local scene data refers to scene data newly added to the current frame relative to the historical frame, and the scene change data refers to scene data whose state has changed relative to the historical frame.

[0095] In a possible implementation, the scene data includes the local scene data and the scene change data; and obtaining the intermediate rendering result of the current frame includes:

[0096] The scene is reconstructed based on the local scene data and the scene data of the historical frame to obtain the reconstructed scene of the current frame; the reconstructed scene of the current frame is updated based on the scene change data; and the rendering intermediate result of the current frame is obtained based on the updated reconstructed scene of the current frame.

[0097] That is, when the scene data of the current frame overlaps with the scene data of the historical frame, and the scene data of the historical frame has been sent to the client, the client can reuse the scene data of the historical frame when reconstructing the scene, so as to combine the scene data of the historical frame and the newly added scene data of the current frame to render and generate the reconstructed scene of the current frame.

[0098] In a possible implementation, the method is applied to a client; before decoding the code stream to obtain scene data of the current frame in the three-dimensional virtual scene, the method further includes:

[0099] Scene interaction information is sent to a target server, where the scene interaction information indicates an operation triggered by an operator in the three-dimensional virtual scene and a position of the operator in the three-dimensional virtual scene.

[0100] In a possible implementation, before sending the scene interaction information to the target server, the method further includes:

[0101] Obtain the operating mode of the three-dimensional virtual scene; if the operating mode is end-cloud collaborative operation, execute the step of sending scene interaction information to the target server in the cloud platform.

[0102] That is, when the client interacts with the cloud platform, the client operator can select the operation mode of the three-dimensional virtual scene in the client, and when the selected operation mode is the end-cloud collaborative mode, the client sends scene interaction information to the target server, so that the target server determines the scene data of the current frame based on the scene interaction information, thereby executing the above encoding method.

[0103] In a possible implementation, the target server is a server in the cloud platform having the shortest communication distance with the client.

[0104] Since the communication distance between the target server and the client is the shortest, the communication delay between the target server and the client is the shortest and the communication performance is better.

[0105] Of course, the target server may also be a server located near the client on the communication connection between the cloud platform and the client, such as a proximal server or a local server of the client. This application does not impose any restrictions on this.

[0106] In a third aspect, a coding device is provided, wherein the coding device has the function of implementing the coding method described in the first aspect. The coding device includes at least one module, wherein the at least one module is configured to implement the coding method described in the first aspect.

[0107] In a fourth aspect, a decoding device is provided, wherein the decoding device has the function of implementing the decoding method described in the second aspect. The decoding device includes at least one module, wherein the at least one module is used to implement the decoding method described in the second aspect.

[0108] In a fifth aspect, an encoding device is provided, comprising one or more processors and a memory, wherein the memory is configured to store a computer program for executing the encoding method provided in the first aspect. The one or more processors are configured to execute the computer program stored in the memory to implement the encoding method described in the first aspect.

[0109] In a possible implementation, the encoding device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0110] In a sixth aspect, a decoding device is provided, comprising one or more processors and a memory, wherein the memory is configured to store a computer program for executing the decoding method provided in the second aspect. The one or more processors are configured to execute the computer program stored in the memory to implement the decoding method described in the second aspect.

[0111] In a possible implementation, the decoding device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0112] In the seventh aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the encoding method described in the first aspect or the decoding method described in the second aspect.

[0113] In an eighth aspect, a computer program product is provided, comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the steps of the encoding method described in the first aspect, or the steps of the decoding method described in the second aspect. Alternatively, a computer program is provided that, when executed on a computer or processor, causes the computer or processor to perform the steps of the encoding method described in the first aspect, or the steps of the decoding method described in the second aspect.

[0114] In the ninth aspect, a coding and decoding system is provided, which includes a coding device and a decoding device, the coding device is used to implement the steps of the coding method described in the first aspect above, and the decoding device is used to implement the steps of the decoding method described in the second aspect above.

[0115] In a tenth aspect, an encoded code stream is provided, wherein the code stream is generated according to the encoding method described in the first aspect.

[0116] In an eleventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a code stream generated according to the encoding method described in the first aspect.

[0117] In a twelfth aspect, a device for storing a code stream is provided, the device comprising: a receiver and at least one storage medium, the receiver being used to receive a code stream generated according to the encoding method described in the first aspect above, and the at least one storage medium being used to store the code stream.

[0118] In the thirteenth aspect, a device for transmitting a code stream is provided, the device comprising: a transmitter and a receiver, the receiver being used to receive the code stream generated by the encoding method described in the first aspect above, and the transmitter being used to send the code stream to an end-side device via a transmission medium.

[0119] In a fourteenth aspect, a device for transmitting a code stream is provided, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the code stream generated according to the encoding method described in the first aspect above, the transmitter being used to obtain the code stream from the storage medium and send the code stream to an end-side device via a transmission medium.

[0120] In a fifteenth aspect, a system for distributing code streams is provided, the system comprising: at least one storage medium for storing code streams generated according to the encoding method described in the first aspect above, a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0121] The technical effects obtained in the above-mentioned third to fifteenth aspects are similar to the technical effects obtained by the corresponding technical means in the first and second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] FIG1 is a schematic diagram of a rendering structure under a single virtual viewpoint provided in an embodiment of the present application;

[0123] FIG2 is a schematic diagram of dividing a surface of a three-dimensional model into multiple facets according to an embodiment of the present application;

[0124] FIG3 is a schematic diagram of a correspondence between pixels and bins provided in an embodiment of the present application;

[0125] FIG4 is a schematic diagram of a cloud rendering process provided by an embodiment of the present application;

[0126] FIG5 is a schematic diagram of an implementation environment of a coding and decoding method provided in an embodiment of the present application;

[0127] FIG6 is a schematic diagram of the structure of a client provided in an embodiment of the present application;

[0128] FIG7 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0129] FIG8 is a schematic diagram of determining a matching block search range according to an embodiment of the present application;

[0130] FIG9 is a schematic flow chart of a decoding method provided in an embodiment of the present application;

[0131] FIG10 is a schematic diagram of a coding and decoding process of a terminal-cloud collaboration provided in an embodiment of the present application;

[0132] FIG11 is a schematic diagram of a game mode switching process provided by an embodiment of the present application;

[0133] FIG12 is a schematic diagram of an edge-cloud collaborative encoding and decoding process provided in an embodiment of the present application;

[0134] FIG13 is a schematic diagram of an encoding and decoding process of a device-edge-cloud collaboration provided by an embodiment of the present application;

[0135] FIG14 is a schematic structural diagram of an encoding device provided in an embodiment of the present application;

[0136] FIG15 is a schematic structural diagram of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0137] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0138] For ease of understanding, before explaining in detail the encoding and decoding method provided in the embodiments of the present application, the terminology, application scenarios, and implementation environment involved in the embodiments of the present application are first introduced.

[0139] First, the terms involved in the embodiments of the present application are introduced.

[0140] 1. Facet

[0141] A facet, also known as a patch, is the smallest planar unit in two- or three-dimensional space. Rendering often requires dividing a model into countless tiny planes, called facets. These facets can be any polygon, with triangles and quadrilaterals being common. The intersection of the edges of these facets is their vertex.

[0142] 2. Pixels

[0143] The screens we typically see are composed of individual pixels. The red, green, and blue (RGB) values ​​of a pixel are determined by the color of the bin that it covers. The meaning of the bins covered by a pixel will be explained in detail later, so I won't go into detail here.

[0144] 3. Graphics rendering engine

[0145] A graphics rendering engine is software that generates realistic or stylized two-dimensional images based on a set of data describing a three-dimensional scene. It receives geometric, texture, and lighting data from an application and, through a series of algorithms and computational steps, ultimately produces a two-dimensional image that conforms to human visual perception. These rendered images are widely used in video games, simulators, film special effects, design visualization, and other fields.

[0146] In an embodiment of the present application, a graphics rendering engine may be deployed on a client and / or a cloud platform (also referred to as a cloud side or cloud end) to render the scene data of the current frame to obtain an intermediate rendering result and a rendered image of the current frame. When the graphics rendering engine is deployed on a cloud platform, the graphics rendering engine may be implemented by a computing device cluster consisting of one or more computing devices.

[0147] 4. Rendering intermediate results

[0148] Intermediate rendering results are intermediate data generated by the graphics rendering engine during the process of generating images / videos to be displayed. These include, but are not limited to, computer graphic motion vectors (CGMVs) generated using computer graphics methods, low-quality rendering results, position maps, normal maps, albedo maps, specular intensity maps, mesh identifiers (Mesh IDs), material identifiers (Material IDs), and depth maps.

[0149] CGMV, also known as Graphics Motion Vector, is data generated by the graphics rendering engine that describes the correspondence between pixels in any two rendered frames. In CGMV, each pixel is associated with a motion vector (MV), which describes the change in the pixel's position within the image sequence. Low-quality renderings typically refer to renderings generated early in the rendering process. They are of lower quality and may exhibit issues such as aliasing and color banding. Because low-quality renderings are typically generated much faster than the final high-quality renderings, developers use them to quickly review the approximate effects of the scene for preliminary adjustments and optimizations. A position map, also known as a depth map or Z-buffer, records the position of each pixel in 3D space in the current frame's rendered image. During the rendering process, the position map is used to determine pixel occlusion relationships, i.e., which pixels should be occluded by other pixels. A normal map stores surface normal directions and surface detail information. Storing surface details in the normal map allows for the creation of uneven surfaces during rendering, enhancing the realism of objects. An albedo map stores basic color information for objects. During rendering, the albedo map determines the base color of an object, enabling the creation of textures and color variations in materials. The specular intensity map stores information about the intensity of light reflected from an object's surface. This allows for highlight effects on the object's surface during rendering, enhancing its gloss and texture.

[0150] The intermediate results in the above examples are all generated by the graphics rendering engine during the rendering process. These intermediate rendering results can assist the graphics rendering engine in achieving more advanced rendering effects, thereby improving rendering efficiency and rendering quality.

[0151] 5. Rendering parameters

[0152] Rendering parameters are inputs to the graphics rendering engine, and together they determine the quality and effect of the final rendered image. In practice, these parameters need to be set and adjusted according to specific needs and scenarios to achieve the best rendering results.

[0153] Among them, rendering parameters include but are not limited to: basic rendering parameters (such as renderer type, output size and format, anti-aliasing, etc.), light source parameters (such as light source position and color vector, lighting model, etc.), observer and scene parameters (such as observer position vector, position coordinates of scene objects, etc.), texture and material parameters (such as texture sampling method, material properties, etc.), animation and motion parameters (motion trajectory of moving objects, skeletal animation parameters, etc.), and other advanced parameters (such as global illumination, ambient occlusion, depth of field and motion blur, etc.).

[0154] 6. Game assets

[0155] Game assets broadly refer to all artwork used to construct game scenes. These assets are the various visual and auditory elements created by game developers through professional art design and production techniques. Game assets can be divided into art assets and audio assets based on their functions and forms of expression. Art assets include materials, textures, meshes, albedo, animations, etc., while audio assets include music and sound effects.

[0156] A mesh is the fundamental data structure used to describe static scenes and dynamic objects in games. It consists of at least a set of vertices defining a three-dimensional (3D) space, a set of triangles connecting these vertices, and a series of information describing vertex properties, such as normals, colors, and texture mapping coordinates (also known as UV coordinates, where U stands for horizontal and V for vertical). These triangles and vertices form the surface of the object represented by the mesh. Graphics processing hardware in computers uses this information to render complex object structures. In practice, a large-scale game typically contains tens of thousands of triangular meshes. Graphics processing software and hardware are much more efficient when processing triangles assembled into meshes than when processing individual triangles. This is because the computer processes the vertices of a triangle mesh, nearly all of which are shared by multiple triangles, and only needs to be processed once. In contrast, each vertex of a single triangle must be processed individually, potentially resulting in significant duplication.

[0157] Just as real-world objects have different materials, different objects in game scenes also need to exhibit different material qualities. This is achieved by defining material information for the object's surface. Material information typically includes the amount of each color reflected by each facet under ambient light, diffuse light, and specular light, as well as the shininess (i.e., the degree to which each facet scatters the specular light). Furthermore, to add more detail to objects, each vertex needs to be colored. However, since realistic objects have countless vertices, defining color and other vertex attributes for each vertex is very expensive. Therefore, graphics programmers tend to use textures to color the surface of objects. Think of a texture as an image, with each vertex of an object having a unique texture coordinate that indicates which point in the image that vertex corresponds to, allowing the texture to be applied to the object's surface. In addition to images, textures can also store a variety of data that is fed into the renderer during the rendering phase.

[0158] In game assets, albedo color describes the underlying color information of an object. When light shines on an object's surface, the diffuse reflection coefficients of the three RGB colors are generated. Albedo color is typically a three-channel RGB texture. Animation controls the behavior of characters and objects in the scene. Common examples include skeletal mesh animation and particle animation. Music refers to the sound effects and background music used during game play.

[0159] 7. Device-edge cloud

[0160] "End" refers to the client, that is, the device used by the user, such as a mobile phone, tablet, personal computer (PC), etc.; "Edge" usually refers to the server that is physically close to the client, usually some edge servers or local servers; "Cloud" refers to the server or server cluster deployed in the cloud, whose location is relatively fixed.

[0161] 8. Potentially Visible Set

[0162] A common concept in computer graphics refers to the objects visible from the current player's perspective (i.e., the observer's perspective) during the rendering process, that is, the set of objects that need to be rendered by the rendering pipeline. If the potentially visible set is not calculated, the graphics rendering engine will render unnecessary invisible objects, thus affecting rendering performance.

[0163] In practical applications, to determine the potentially visible set, invisible objects need to be removed from all objects in the scene. This process is called occlusion culling. Common occlusion culling methods include rendering pipeline culling, online culling (such as hardware occlusion query technology, software occlusion query technology, hierarchical depth buffer, and frame difference culling technology), and offline culling (such as precomputed occlusion culling technology).

[0164] Secondly, considering that the encoding and decoding method provided in the embodiment of the present application involves image rendering, the relationship between the rendering structure, facets, vertices and pixels involved in the image rendering process is first introduced here.

[0165] Please refer to FIG1 , which is a schematic diagram of a rendering structure under a single virtual viewpoint provided by an embodiment of the present application. The rendering structure at least includes a virtual viewpoint 101 , a virtual viewing plane 102 , a three-dimensional model 103 , and a virtual light source 104 .

[0166] Virtual viewpoint 101 is a simulated human eye or eyes in space, used to perceive three-dimensional structures. Virtual viewpoint 101 is categorized as monocular, binocular, or multi-viewpoint, depending on the number of viewpoints it includes. Binocular or multi-viewpoint refers to capturing two or more images from two or more different viewpoints, and then reconstructing a three-dimensional model from these two or more images.

[0167] The virtual viewing plane 102 is a display screen simulated in space. The construction of the virtual viewing plane 102 depends on two factors: the distance between the virtual viewpoint 101 and the virtual viewing plane 102 and the screen resolution of the virtual viewing plane 102 .

[0168] The distance between the virtual viewpoint 101 and the virtual viewing plane 102 refers to the vertical distance from the virtual viewpoint 101 to the virtual viewing plane 102 , and the vertical distance can be set according to different requirements.

[0169] The screen resolution of the virtual viewing plane 102 refers to the number of pixels included in the virtual viewing plane 102. In other words, the virtual viewing plane 102 includes one or more pixels (in FIG. 1 , the virtual viewing plane 102 is schematically illustrated as including 9 pixels).

[0170] The display screen of the user terminal (also referred to as the client) corresponding to the virtual viewing plane 102 is used to output the rendering result. The screen resolution of the display screen may be equal to or different from the screen resolution of the virtual viewing plane 102. When the screen resolution of the display screen is equal to the screen resolution of the virtual viewing plane 102, the rendering result on the virtual viewing plane 102 is output to the display screen at a 1:1 ratio. When the screen resolution of the display screen is different from the screen resolution of the virtual viewing plane 102, the rendering result on the virtual viewing plane 102 is output to the display screen at a certain ratio.

[0171] Typically, a rendering application includes at least one three-dimensional model 103, that is, the at least one three-dimensional model 103 is reconstructed by software or a program, and which three-dimensional models 103 are included in the rendering result on the virtual viewing plane 102 is determined by the relative position between the virtual viewpoint 101 (or the camera pose / user perspective) and the three-dimensional model 103.

[0172] In some embodiments, before rendering the at least one 3D model 103 by a graphics rendering engine, the at least one 3D model 103 needs to be divided into a plurality of facets, wherein the size and shape of each facet in the plurality of facets may be the same or different.

[0173] For example, please refer to Figure 2, which is a schematic diagram of an embodiment of the present application providing a method of dividing a face of a three-dimensional model 103 into multiple facets. In Figure 2, a face of the three-dimensional model 103 is divided into six triangular facets of different sizes, namely facets 1-6. Among them, the vertices of facet 1 are D0, D1, and D2, the vertices of facet 2 are D0, D2, and D3, the vertices of facet 3 are D0, D3, and D4, the vertices of facet 4 are D0, D4, and D5, the vertices of facet 5 are D0, D5, and D6, and the vertices of facet 6 are D0, D1, and D6.

[0174] For example, please refer to Figure 3, which is a schematic diagram of a pixel-surface correspondence provided in an embodiment of the present application. In Figure 3, the thicker solid box represents the projection area of ​​a pixel included in the virtual viewing plane 102 on the three-dimensional model 103. The projection area of ​​the pixel on the three-dimensional model 103 covers part of the area of ​​6 triangular surfels.

[0175] In one possible implementation, the projection area of ​​a pixel on the three-dimensional model 103 may cover multiple bins or only one bin. When the projection area of ​​a pixel on the three-dimensional model 103 covers only one bin, it may cover the entire area of ​​the bin or a portion of the area of ​​the bin.

[0176] Virtual light sources 104 are simulated light sources within a space, used to generate the lighting environment within the space. Virtual light sources 104 can be any of the following: point light sources, surface light sources, and line light sources. A space can include one or more virtual light sources 104. When a space includes multiple virtual light sources 104, the types of these virtual light sources 104 can be the same or different.

[0177] Next, the application scenarios involved in the embodiments of the present application are introduced.

[0178] With advances in technology and the development of cloud computing, cloud rendering has become a vital tool in industries such as gaming, design, architecture, and film and television. As shown in Figure 4, cloud rendering involves using a high-quality graphics rendering engine on a cloud platform to render images based on 3D models and other related data. This image is then encoded into a bitstream, which is then transmitted over the network to the client, where it is parsed and displayed. This not only reduces client hardware and maintenance costs, but also leverages the powerful performance and elastic scalability of the cloud platform to improve rendering efficiency and quality.

[0179] Taking the gaming industry as an example, cloud gaming is an online gaming service based on cloud computing. Game logic and game assets are stored on cloud servers, allowing players to control the game remotely over the network without having to download and install the game on a local client. In cloud gaming mode, the cloud platform receives player operation information transmitted by the client, renders the game scene for the current frame, and transmits the rendered image to the client via the network for display.

[0180] Compared with image rendering on the client side, transmitting rendered images between the cloud platform and the client requires a considerable amount of bitrate bandwidth and server computing resources, which is also one of the main costs for cloud gaming manufacturers. Therefore, it is necessary to reduce the bitrate bandwidth and computing resources in the cloud gaming mode.

[0181] In cloud gaming mode, the cloud platform can use a standard video encoder to encode the rendered image and transmit the encoded code stream to the client via the network. Similarly, the client uses a standard video decoder to decode the code stream transmitted by the cloud platform to obtain a reconstructed image. In this way, the cloud gaming mode based on video streaming to transmit rendered images enables games to be run on various clients.

[0182] Among them, the standard video codec can be implemented according to the industry video compression standard, and the video compression standard includes but is not limited to: advanced video coding (AVC, also known as H.264) standard, high efficiency video coding (HEVC, also known as H.265) standard, versatile video coding (VCC, also known as H.266) standard, audio and video coding standard (AVS) formulated by China, and extensions of the above standards. Among them, the audio and video coding standard formulated by China includes the second-generation audio and video coding standard (also known as AVS2), and the third-generation audio and video coding standard (also known as AVS3) optimized and improved on the basis of AVS2. The encoding and decoding method provided in the embodiment of the present application is not limited to any specific encoding and decoding standard.

[0183] However, with the development of terminal hardware technology and the evolution of device form factors, some clients now have image rendering capabilities, and can independently or partially complete rendering tasks based on scene data to generate corresponding rendered images. For clients with strong image rendering capabilities, they can run the game locally and independently complete scene rendering and screen display without relying on the computing resources of the cloud platform. For clients with slightly weaker image rendering capabilities, they may not be able to run the entire game independently and achieve the same effect and experience as cloud gaming mode, but they can still use the client's own image rendering capabilities to perform partial rendering operations, thereby reducing the bitrate bandwidth when the cloud platform transmits the rendered image.

[0184] This shows that in the cloud gaming mode, the client's image rendering capabilities are not fully utilized. Reasonable use of the client's image rendering capabilities can reduce the bitrate bandwidth when encoding and transmitting rendered images from the cloud platform.

[0185] Unlike natural video, game images are rendered using computer graphics technology. During the rendering process, a large amount of prior information (intermediate rendering results) is stored. Leveraging these intermediate rendering results to assist cloud platforms in encoding and transmitting rendered images (e.g., game images) and assisting clients in parsing the transmitted bitstream from the cloud platform to reconstruct the rendered image can effectively reduce the bitrate and computational overhead required to encode and render the image.

[0186] It's important to note that codec consistency can only be guaranteed if the cloud platform and the client achieve the same intermediate rendering results during the rendering process. Using a client with powerful image rendering capabilities to run a full game requires the client to have the complete game assets. This means the client must pre-download and install all game assets, which raises the barrier to entry and reduces the gaming experience. While running the full game on the cloud platform and then transmitting the game screen to the client allows for the storage of complete game assets on the cloud platform, the bandwidth required to transmit the game screen on the cloud platform is high, placing certain demands on the bandwidth resources between the cloud platform and the client.

[0187] In some related technologies, if the client network environment includes a client with sufficient graphics processing resources to remotely execute a game, a game server stack can be downloaded by the client, and the game server can be executed locally on the client. However, while the game server stack is being downloaded to the client, the game can be executed by the remote server, and the rendered game screen can be streamed to the client.

[0188] In other words, the game can be executed remotely while the game assets are downloaded for local execution. That is, the game can initially run in the cloud, which renders the game screen and transmits the encoded game screen stream to the client for decoding and playback. At the same time, the game logic and assets are downloaded to the client in the background. When the game is launched on the client, the game is also launched and executed remotely on the cloud side. The game logic and assets are simultaneously transferred from the cloud game container to the container in the game server stack on the high-performance client. Once a sufficient amount of game logic and / or assets are downloaded to the game server stack, the game can be run locally using one or more GPUs and local central processing units (CPUs) on the high-performance client.

[0189] As can be seen, the relevant technology first runs cloud gaming mode, using video streaming to transmit the encoded game screen to the client, and also transmits the game assets to the client. After the client downloads a sufficient amount of game assets, it switches to client gaming mode, where the client independently performs image rendering and display, without the need for interaction with the cloud.

[0190] However, as mentioned above, the cloud gaming mode has a high bitrate bandwidth and does not utilize the client's own image rendering capabilities. The client-side gaming mode places relatively high demands on the client's image rendering capabilities, and ordinary clients cannot render images with effects similar to those rendered on the cloud platform.

[0191] Based on this, an embodiment of the present application provides a coding and decoding method, in which an end-cloud collaborative mode is run between the cloud platform and the client. In this mode, the cloud platform renders a rendered image of the current frame, and based on the scene data of the current frame, encodes the rendered image of the current frame into a code stream, and at the same time encodes the scene data of the current frame into the code stream. The client receives the code stream transmitted by the cloud platform, decodes it to obtain the scene data of the current frame, and determines the rendered image reconstructed for the current frame based on the scene data. In this way, the client does not need to download all the data of the complete three-dimensional virtual scene in advance, but only needs to obtain the scene data of the current frame during the process of end-cloud collaborative rendering of the three-dimensional virtual scene, which lowers the threshold for the client to run the three-dimensional virtual scene and improves the user experience.

[0192] Taking a game scene as an example, a 3D virtual scene is constructed based on the game's global game assets, including the game's global environment map and the virtual objects and characters within it. The scene data for the current frame refers to the game assets required to render the game screen for that frame, and these game assets are a subset of the global game assets.

[0193] It should be noted that when introducing the application scenarios of the embodiments of the present application, only the game scenario is used as an example. In actual applications, the encoding and decoding method provided by the embodiments of the present application can be applied not only to the game field, but also to other fields related to image rendering technology, such as virtual reality (VR) field, augmented reality (AR) field, digital human field, etc., and the embodiments of the present application are not limited to this.

[0194] Next, the implementation environment involved in the embodiments of the present application is introduced.

[0195] Please refer to Figure 5, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. Source device 10 can generate an encoded image, i.e., a code stream. Therefore, source device 10 can also be referred to as an encoding device. Destination device 20 can decode the code stream generated by source device 10. Therefore, destination device 20 can also be referred to as a decoding device. Link 30 can receive the encoded image generated by source device 10 and transmit it to destination device 20. Storage device 40 can receive the encoded image generated by source device 10 and store it. In this case, destination device 20 can directly obtain the encoded image from storage device 40. Alternatively, storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded image generated by source device 10. In this case, destination device 20 can stream or download the encoded image stored by storage device 40.

[0196] The source device 10 and the destination device 20 may each include one or more processors and a memory coupled to the one or more processors. The memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of computer-accessible instructions or data structures. For example, the source device 10 may be a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud computing service center. The destination device 20 may include a mobile phone, a smartphone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, a smart car computer, a smart TV, a smart speaker, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called "smart" phone, a television, a camera, a display device, a digital media player, a video game console, an in-vehicle computer, or the like.

[0197] Link 30 may include one or more media or devices capable of transmitting encoded images from source device 10 to destination device 20. In one possible implementation, link 30 may include one or more communication media that enable source device 10 to send encoded images directly to destination device 20 in real time. In an embodiment of the present application, source device 10 may modulate the encoded images based on a communication standard, such as a wireless communication protocol, and may transmit the modulated images to destination device 20. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from source device 10 to destination device 20, although this embodiment of the present application does not specifically limit this.

[0198] In one possible implementation, the storage device 40 may store the received encoded image sent by the source device 10, and the destination device 20 may directly obtain the encoded image from the storage device 40. Under such conditions, the storage device 40 may include any of a variety of distributed or locally accessible data storage media, for example, any of the various distributed or locally accessible data storage media may be a hard disk drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing a code stream.

[0199] In one possible implementation, storage device 40 may correspond to a file server or another intermediate storage device that can store the codestream generated by source device 10. Destination device 20 may stream or download the images stored on storage device 40. The file server may be any type of server capable of storing and transmitting encoded images to destination device 20. In one possible implementation, the file server may include a network server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Destination device 20 may obtain the encoded images via any standard data connection, including an internet connection. Any standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for obtaining encoded images stored on a file server. The transmission of the encoded images from storage device 40 may be streaming, downloading, or a combination of both.

[0200] The implementation environment shown in FIG5 is only one possible implementation method, and the technology of the embodiment of the present application is applicable not only to the source device 10 that can encode images and the destination device 20 that can decode encoded images shown in FIG5 , but also to other devices that can encode images and decode code streams. The embodiment of the present application does not make specific limitations on this.

[0201] In the implementation shown in FIG5 , source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, output interface 140 may include a modem and / or a transmitter, where the transmitter may also be referred to as a transmitter. Data source 120 may include an image capture device (e.g., a camera), an archive containing previously captured images, a feed interface for receiving images from an image content provider, and / or a computer graphics system for generating images, or a combination of these sources of images.

[0202] The data source 120 may send an image to the encoder 100, and the encoder 100 may encode the image received from the data source 120 to generate an encoded image. The encoder may send the encoded image to an output interface. In some embodiments, the source device 10 directly sends the encoded image to the destination device 20 via the output interface 140. In other embodiments, the encoded image may also be stored on the storage device 40 for later retrieval by the destination device 20 for decoding and / or display.

[0203] In the implementation environment shown in FIG5 , the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive encoded images via the link 30 and / or from the storage device 40, and then transmit the encoded images to the decoder 200. The decoder 200 may decode the received encoded images to obtain decoded images. The decoder may transmit the decoded images to the display device 220. The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20. Generally, the display device 220 displays the decoded images. The display device 220 may be any of a variety of types of display devices, for example, the display device 220 may be a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0204] Although not shown in FIG5 , in some aspects, the encoder 100 and decoder 200 can be integrated with an encoder and decoder, respectively, and can include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or in separate data streams. In some embodiments, the MUX-DEMUX units can conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP), if applicable.

[0205] The encoder 100 and the decoder 200 can each be any of the following circuits: one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology of the embodiments of the present application is implemented in part by software, the device can store instructions for the software in a suitable non-volatile computer-readable storage medium, and can use one or more processors to execute the instructions in hardware to implement the technology of the embodiments of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors. Each of the encoder 100 and the decoder 200 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined encoder / decoder (encoder / decoder) in the corresponding device.

[0206] Embodiments of the present application may generally refer to encoder 100 as "signaling" or "sending" certain information to another device, such as decoder 200. The terms "signaling" or "sending" may generally refer to the transmission of syntax elements and / or other data used to decode a compressed image. This transmission may occur in real time or near real time. Alternatively, this communication may occur over time, such as when the syntax elements are stored in the encoded bitstream to a computer-readable storage medium during encoding, and the decoding device may then retrieve the syntax elements at any time after they are stored to this medium.

[0207] The encoding and decoding methods provided in the embodiments of the present application can be applied to a variety of scenarios. In various scenarios, the images encoded and decoded can be images included in image files or images included in video files. It should be noted that, in conjunction with the implementation environment shown in Figure 5, any of the encoding methods below can be executed by the encoder 100 in the source device 10, and the encoder 100 is implemented by software, hardware, or a combination of both to become part or all of the cloud platform in the embodiments of the present application. Any of the decoding methods below can be executed by the decoder 200 in the destination device 20, and the decoder is implemented by software, hardware, or a combination of both to become part or all of the client in the embodiments of the present application.

[0208] For example, a cloud platform can be a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud computing service center. The cloud platform can implement the encoding method provided in the embodiment of Figure 7 below.

[0209] Please refer to Figure 6, which is a schematic diagram of the structure of a client according to an embodiment of the present application. The client includes at least one processor 601, a communication bus 602, a memory 603, and at least one communication interface 604. The client has certain image rendering capabilities and can render scene data to obtain intermediate rendering results.

[0210] The processor 601 may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0211] Communication bus 602 is used to transmit information between the above components. Communication bus 602 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0212] The memory 603 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 603 may exist independently and be connected to the processor 601 via the communication bus 602. The memory 603 may also be integrated with the processor 601.

[0213] The communication interface 604 uses any device such as a transceiver to communicate with other devices or communication networks. The communication interface 604 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0214] In a specific implementation, as an embodiment, the processor 601 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG6 .

[0215] In a specific implementation, as an example, the client may include multiple processors, such as processor 601 and processor 605 shown in FIG6 . Each of these processors may be a single-core processor or a multi-core processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0216] In a specific implementation, as an embodiment, the client may further include an output device 606 and an input device 607. The output device 606 communicates with the processor 601 and can display information in a variety of ways. For example, the output device 606 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 607 communicates with the processor 601 and can receive user input in a variety of ways. For example, the input device 607 can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0217] In some embodiments, the memory 603 is used to store program code 610 for executing the solution of the present application, and the processor 601 can execute the program code 610 stored in the memory 603. The program code 610 may include one or more software modules, and the client can implement the decoding method provided in the embodiment of Figure 9 below through the processor 601 and the program code 610 in the memory 603.

[0218] It should be noted that the application scenarios, coding and decoding standards, and implementation environments described in the above examples are intended to more clearly illustrate the coding and decoding scheme of the embodiments of the present application, and do not constitute a limitation on the coding and decoding scheme of the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios, new coding and decoding standards, and the evolution of the implementation environment, the coding and decoding scheme of the embodiments of the present application is also applicable to similar technical problems.

[0219] Next, the encoding method and decoding method provided in the embodiments of the present application are described in detail respectively.

[0220] In an embodiment of the present application, the encoding method can be applied to a cloud platform, and the decoding method can be applied to a client. The cloud platform includes one or more target servers, and the one or more target servers can communicate with the client to execute the encoding method provided in the embodiment of the present application. Taking into account the problem of communication delay, the target server can be the server with the shortest communication distance with the client in the cloud platform, and the encoding method provided in the embodiment of the present application is executed by the target server in the cloud platform, and the encoded code stream is transmitted to the client through the network, or the target server transmits the encoded code stream to other transfer devices in the network domain where the client is located, such as a local server, and the local server can communicate with the client.

[0221] In some embodiments, the encoding method provided by the embodiments of the present application can also be executed by a local server located within the same local area network as the client. The local server obtains the scene data and rendered image of the current frame from the cloud platform, and based on the scene data of the current frame, encodes the rendered image of the current frame into a bitstream, and transmits the encoded bitstream to the client via the network. This significantly shortens the communication delay between the client and the server.

[0222] For game scenes, the cloud platform not only includes all relevant 3D models of the target game scenes, but also includes information such as the game logic for running the target game. This means the cloud platform can independently run the target game and, using traditional cloud gaming mode, send the rendered game screen to the client via video streaming, allowing the client to decode and reconstruct the screen. The target game is any game launched by the client and selected to run in cloud gaming mode or end-cloud collaborative gaming mode.

[0223] Based on the aforementioned cloud gaming model, in the encoding method described in the embodiment of this application, the cloud platform sends the scene data of the current frame to the client and, based on the scene data of the current frame, encodes the rendered image of the current frame. Compared to traditional video streaming methods, the encoding and decoding scheme of the embodiment of this application can significantly reduce the bitrate bandwidth when transmitting rendered images between the cloud platform and the client.

[0224] Please refer to Figure 7, which is a schematic flow chart of an encoding method provided in an embodiment of the present application. This encoding method is illustrated by applying it to a target server in a cloud platform. The target server is the server in the cloud platform with the shortest communication distance to the client. The encoding method can be specifically applied to an encoder in the target server. The encoding method includes the following steps.

[0225] Step 701: Acquire scene data of the current frame in the three-dimensional virtual scene.

[0226] Among them, the scene data is all or part of the data required to construct the current frame scene (that is, the three-dimensional virtual scene corresponding to the current frame), such as the mesh data and albedo data required to construct the current frame scene. The scene data can also include material data, texture data, animation data, music data, etc. for constructing the current frame scene. The embodiments of the present application do not limit this.

[0227] It should be noted that the current frame scene may be the above-mentioned three-dimensional virtual scene, or may be a subset or sub-scene of the above-mentioned three-dimensional virtual scene.

[0228] Taking the game scene as an example, the three-dimensional virtual scene in step 701 can be all the three-dimensional virtual scenes involved in the target game, which corresponds to the global game scene of the target game, including the three-dimensional models corresponding to all virtual objects and / or characters in the global game map; and the current frame scene can be the game scene under a certain perspective / camera posture, which corresponds to the local game scene of the target game, and only includes the three-dimensional models corresponding to all virtual objects and / or characters in the local game map.

[0229] In some embodiments, the scene data for the current frame may be global scene data for the current frame, i.e., data for all three-dimensional models in the current frame scene. In this case, after the target server sends the scene data for the current frame to the client, the client can directly reconstruct the current frame scene based on the scene data of the current frame without referencing scene data from other frames.

[0230] In one possible implementation, if the target server has started communicating with the client before executing the above step 701 and has sent scene data of part of the historical frame to the client, since the historical frame and the current frame are continuous in time sequence, they may involve the same three-dimensional virtual scene, and the scene data of the historical frame and the current frame may partially overlap. At this time, when the target server executes the above step 701, the scene data of the current frame obtained may only include the scene data newly added to the current frame relative to the historical frame.

[0231] In this case, the scene data for the current frame obtained by the target server is the local scene data of the current frame, which can be the data of a portion of the 3D model in the current frame scene. In this case, after the target server sends the scene data of the current frame to the client, the client needs to combine it with the scene data of the previous frames it has received to reconstruct the current frame scene.

[0232] In some embodiments, the scene data of the current frame in the embodiments of the present application may include local scene data and / or scene change data. The local scene data refers to the scene data newly added to the current frame relative to the historical frame, and the scene change data refers to the scene data whose state has changed relative to the historical frame.

[0233] That is, the scene data of the current frame may include only the local scene data of the current frame, or only the scene change data of the current frame, or may include both the local scene data and the scene change data of the current frame. This embodiment of the present application does not impose any restrictions on this.

[0234] Among them, the three-dimensional virtual scene corresponding to the current frame may include multiple two-dimensional / three-dimensional models, and the scene change data of the current frame is used to describe the state change of at least one three-dimensional model in the current frame relative to the historical frame, or the change of part of the grid in the at least one three-dimensional model.

[0235] As an example, when the scene data of the current frame includes mesh data, the scene change data includes vertex attribute change data of the mesh, such as vertex coordinate positions and normals, etc. Based on the scene change data, the state of the corresponding mesh / 3D model in the scene can be adjusted.

[0236] In some embodiments, when the target server communicates with the client, the process of the target server executing the above step 701 can be: receiving scene interaction information sent by the client, the scene interaction information indicating the operation triggered by the operator in the three-dimensional virtual scene, and the position of the operator in the three-dimensional virtual scene; based on the scene interaction information, determining the scene data of the current frame.

[0237] In other words, the scene interaction information indicates the operator's movement in the three-dimensional virtual scene and the operations triggered on the relevant three-dimensional models in the three-dimensional virtual scene.

[0238] As an example, in a game scene, the scene interaction information may be the movement information of the game character corresponding to the operator in the three-dimensional virtual scene, as well as the operations triggered by the game character on the relevant three-dimensional models in the three-dimensional virtual scene, such as the destruction, construction, and state modification of the three-dimensional models in the three-dimensional virtual scene. The operations that can be triggered by the game character in the embodiment of the present application are not limited.

[0239] Taking a game scene as an example, the target server in the embodiments of the present application can store all scene data for the target game requested by the client. That is, the target server stores scene data for all three-dimensional virtual scenes involved in the target game. Based on this, the target server determines the scene data for the current frame from the three-dimensional virtual scene corresponding to the target game based on the operator's position in the three-dimensional virtual scene.

[0240] In one possible implementation, the implementation process of determining the scene data of the current frame based on scene interaction information may be as follows: based on the scene interaction information, the scene data within the target range is determined as the scene data of the current frame, and the target range refers to the range with the operator's position in the three-dimensional virtual scene as the center point and extending the target size to the surrounding area.

[0241] The target size may be a preset value, and the embodiment of the present application does not impose any limitation on this value.

[0242] As an example, in a game scene, based on the position of the game character corresponding to the operator in the three-dimensional virtual scene, the scene data corresponding to the game environment within a certain range around the game character in the game map is determined as the scene data of the current frame.

[0243] In another possible implementation, the implementation process of determining the scene data of the current frame based on the scene interaction information may be: based on the scene interaction information, determining the posture of the virtual camera in the three-dimensional virtual scene; based on the posture of the virtual camera, determining the scene data within the visible range of the virtual camera as the scene data of the current frame.

[0244] That is, based on the scene interaction information, the operation triggered by the operator in the three-dimensional virtual scene is determined, the position of the virtual camera in the three-dimensional scene is determined based on the operation, and the scene data within the visible range of the virtual camera is determined as the scene data of the current frame.

[0245] In one possible implementation, considering that the amount of scene data within the visible range of the virtual camera is large, it takes a long time for the target server to encode the scene data or transmit the scene data to the client. Moreover, from the operator's perspective, the scene data within the visible range of the virtual camera, or the scene within the above-mentioned target range, may include data of three-dimensional models visible to the operator at the current position, as well as data of three-dimensional models not visible to the operator at the current position. Based on this, after determining the scene data within the target range or the scene data within the visible range of the virtual camera, the scene data can also be subjected to culling processes such as frustum culling, occlusion culling, and backface culling based on the operator's position in the three-dimensional virtual scene using a grid culling method, and the processed scene data is determined as the scene data of the current frame.

[0246] In some embodiments, when determining the scene data for the current frame in a game scene, the player's current game level can be determined based on scene interaction information for the target game launched by the player. The entire scene data for the current level is then sent to the client during the transmission of the rendered image for the current frame. In this case, the scene data for the current frame represents a portion of the scene data corresponding to the aforementioned three-dimensional virtual scene.

[0247] In a possible implementation, all scene data of the target game may be sent to the client during the transmission of the rendered image of the current frame. In this case, the scene data of the current frame is all scene data of the three-dimensional virtual scene.

[0248] It should be noted that when the target server encodes a large amount of scene data in the current frame, resulting in low encoding efficiency and / or a long time to transmit the code stream to the client, the target server can send the rendered image of the current frame to the client in a traditional video streaming manner.

[0249] Furthermore, after the client receives the scene data of the current game level, or the scene data of the entire target game, transmitted by the target server, the target server and the client still use the virtual reference image provided in the embodiment of the present application to assist in encoding and decoding the rendered image of the current frame to transmit the bit stream, but at this time the bit stream no longer contains the scene data of the current frame.

[0250] Thus, it can be seen that for the scene data of the current frame, the embodiments of the present application do not restrict the coverage of the scene data (or the data volume of the scene data), and can be flexibly selected based on the network bandwidth and latency requirements between the client and the target server. After the target server and the client both have the scene data of the current frame, it can be guaranteed that the target server and the client can render the same intermediate rendering result, thereby generating the same virtual reference image to assist in encoding and decoding the rendered image of the current frame.

[0251] In addition, since the encoding and decoding method provided in the embodiment of the present application requires that both the target server and the client have the scene data of the current frame and both have image rendering capabilities, can render the scene data to obtain a rendering intermediate result, and generate the same virtual reference image based on the same rendering intermediate result, therefore, whether the target server executes the above-mentioned step 701, or whether to use the encoding method provided in the embodiment of the present application to generate the bit stream, needs to be judged in combination with the client's image rendering capabilities, and / or the network bandwidth between the target server and the client, so as to determine whether the target server and the client can use end-cloud collaboration to execute the solution of the embodiment of the present application.

[0252] In some embodiments, when the target server executes the encoding method provided in the embodiments of the present application, it also includes: obtaining the device information of the client, which indicates the image rendering capability of the client; if it is determined based on the device information of the client that the three-dimensional virtual scene requires end-cloud collaboration, then the above step 701 is executed, and the end-cloud collaboration mode is run between the target server and the client.

[0253] It should be noted that the three-dimensional virtual scene requires end-cloud collaborative operation, which means that both the target server and the client in the cloud platform need to build a three-dimensional virtual scene, and both the target server and the client need to have image rendering capabilities and be able to obtain the same rendering intermediate results based on the scene data of the current frame in the three-dimensional virtual scene.

[0254] In other words, the client's image rendering capabilities are determined based on the client's device information. Based on the client's image rendering capabilities, it can be determined whether the client can render the scene data of the current frame to obtain the current frame's intermediate rendering result. Therefore, if the client's image rendering capabilities enable the client to generate the same intermediate rendering result as the target server, the end-cloud collaboration mode is determined to be in operation.

[0255] In one possible implementation, the process of determining whether a three-dimensional virtual scene requires end-cloud collaboration based on the client's device information can be as follows: based on the client's device information, the client's rendering capability level can be determined; based on the client's rendering capability level, whether the three-dimensional virtual scene requires end-cloud collaboration can be determined.

[0256] Among them, the device information includes multiple device parameters of the client device, such as the client's device type, memory and running memory size, GPU model, graphics driver version number, operating system type, etc., which are not limited in this embodiment of the present application.

[0257] As an example, when determining the rendering capability of a client, the various device parameters of the client device can be scored based on the device information of the client and the thresholds corresponding to the various device parameters, and the scores of the various device parameters can be weighted and summed to determine the rendering capability level of the client.

[0258] Among them, the rendering capability level of the client includes the first rendering level, the second rendering level and the third rendering level. The rendering capability level is the first rendering level, which means that the client has strong image rendering capability and can independently render the scene data to obtain the rendered image and rendering intermediate results of the current frame. Its image rendering capability is close to that of the cloud platform; the rendering capability level is the second rendering level, which means that the client has strong image rendering capability but not as good as the cloud platform. The client can render the scene data to obtain the rendering intermediate results of the current frame, but cannot render a rendered image with an effect close to that of the cloud platform; the rendering capability level is the third rendering level, which means that the client has weak image rendering capability or does not have image rendering capability and can only parse the code stream but cannot render the scene data of the current frame to obtain the rendering intermediate results.

[0259] Based on this, if it is determined that the rendering capability level of the client is the second rendering level, the end-cloud collaboration mode is run. Of course, when the rendering capability level of the client is the first rendering level, the end-cloud collaboration mode can also be run, and this embodiment of the application does not limit this.

[0260] It can be seen from this that when the solution of the embodiment of the present application requires that the client also has certain image rendering capabilities, before executing the above step 701, the target server can evaluate the client's image rendering capabilities in combination with the client's device information, and execute the solution of the embodiment of the present application when it is determined that the three-dimensional virtual scene requires the end-cloud to operate in collaboration, so as to avoid the situation where the client's image rendering capabilities are insufficient, resulting in the inability to generate a rendered image reconstructed from the current frame based on the code stream sent by the cloud platform.

[0261] In one possible implementation, when it is determined that the client's image rendering capability is insufficient, for example, when the client's rendering capability level is the third rendering level, the target server can send the rendered image to the client using the traditional video streaming method. That is, when the target server generates the rendered image by executing the following step 702, the rendered image of the current frame is encoded into the bitstream based on the rendered image of the historical frame, and the bitstream is sent to the client.

[0262] In some embodiments, when the target server executes the encoding method provided in the embodiments of the present application, it also includes: obtaining the network bandwidth between the target server and the client; if the three-dimensional virtual scene is not currently running in end-cloud collaboration, and the network bandwidth meets the switching conditions for end-cloud collaboration, then execute the above step 701, and at this time the end-cloud collaboration mode is running between the target server and the client.

[0263] It should be understood that the target server and the client use end-cloud collaboration to encode and decode the rendered image of the current frame, which means that both the target server and the client can obtain the same rendering intermediate result, and use the same rendering intermediate result to encode the rendered image of the current frame, or decode to obtain the reconstructed rendered image of the current frame.

[0264] When a 3D virtual scene operates in a device-cloud collaborative environment, the target server and client need to transmit a rendered image of the current frame encoded with scene data, as well as a bitstream containing the scene data of the current frame. This places certain demands on the network bandwidth between the target server and the client. Therefore, during communication with the client, the target server can detect the network bandwidth between the two in real time and determine whether the current network bandwidth meets the switching conditions for device-cloud collaboration.

[0265] In one possible implementation, if the network bandwidth between the target server and the client is greater than or equal to the bandwidth threshold, it is determined that the network bandwidth meets the operating conditions of the end-cloud collaboration mode, and the end-cloud collaboration mode can be run between the target server and the client, that is, the technical solution provided in the embodiment of the present application is used to encode and decode the rendered image of the current frame.

[0266] In one possible implementation, if the network bandwidth does not meet the operating conditions of the end-cloud collaboration mode, and the scene data of the current frame is stored in the cloud platform, the target server in the cloud platform uses the historical frame image to encode the rendered image of the current frame, and sends the code stream encoded with the rendered image of the current frame to the client.

[0267] In one possible implementation, if the network bandwidth does not meet the operating conditions of the end-cloud collaborative mode, and the client's image rendering capability is strong (for example, the client's rendering capability level is the first rendering level), and the client already has the scene data of the current frame, the client will autonomously render and generate the rendered image of the current frame based on the scene data of the current frame, and send it for presentation.

[0268] Step 702: Encode the scene data into a bitstream.

[0269] Based on the scene data of the current frame, the scene data of the current frame is encoded into the bitstream according to the relevant encoding algorithm. Furthermore, the target server may encode the scene data into the bitstream after obtaining the scene data of the current frame, or may encode the scene data into the bitstream after executing step 703. The present embodiment does not limit the order in which steps 702 and 703 are executed.

[0270] Step 703: Encode the rendered image of the current frame into a bitstream based on the scene data.

[0271] It should be noted that the code stream encoding the rendered image of the current frame and the code stream encoding the scene data of the current frame can be the same code stream or different code streams, and the embodiment of the present application does not impose any limitation on this.

[0272] In a possible implementation, the implementation process of step 703 includes the following steps: obtaining an intermediate rendering result of the current frame, where the intermediate rendering result is obtained by rendering scene data; and encoding the rendered image based on the intermediate rendering result.

[0273] The target server may include a high-quality graphics rendering engine, which renders the scene data of the current frame to obtain a rendered image and an intermediate rendering result of the current frame. Regarding the implementation process of the graphics rendering engine rendering the scene data of the current frame to obtain the intermediate rendering result and the rendered image, reference may be made to relevant prior art, and this embodiment of the application does not provide a detailed description or limitation here.

[0274] The intermediate rendering results are the prior information generated during the rendering of the current frame. They may include one or more of the current frame's CGMV, low-quality rendering results, position map, normal map, albedo map, specular intensity map, bin identifier, material identifier, and depth map. These intermediate rendering results can be used in the encoding process of the rendered image to optimize the encoding process.

[0275] In some embodiments, encoding the rendered image of the current frame includes intra-frame prediction and inter-frame prediction. Taking inter-frame prediction as an example, the process of encoding the rendered image of the current frame includes steps such as blocking, motion estimation, MV prediction, motion compensation, quantization, and entropy coding. Therefore, based on the intermediate rendering results generated during the rendering of the scene data, at least one of the above steps can be optimized to achieve an optimized encoding process for the rendered image of the current frame.

[0276] The purpose of block segmentation is to divide the rendered image into smaller regions so that each region can be processed and encoded separately. For example, a rendered image can be divided into multiple 64×64 coding tree units (CTUs), which can be further divided into smaller coding units (CUs), prediction units (PUs), and transform units (TUs). Motion estimation is the process of finding the best matching block in a previously coded image for each block in the rendered image. This process determines the location of the matching block by comparing the similarity between the current image block and a reference block. For example, if the rendered image of the current frame contains a red block, the encoder searches for the most similar red block in the rendered image of the reference frame and records the location of this red block in the reference frame. This location information is the motion vector (MV). Based on the motion vector and inter-frame prediction method, an estimated value for the current frame can be determined. This process is called motion compensation. The goal of motion compensation is to use information from the coded image to predict the content of the current frame, thereby reducing the amount of information required for encoding. For example, the encoder uses the information of the best matching red block in the reference frame to predict the information of the red block in the rendered image of the current frame. The difference between the predicted value and the actual value is the prediction residual. Quantization is the process of converting the prediction residual into a discrete value, which further reduces the amount of data by reducing the accuracy of the residual signal. Entropy coding is the process of losslessly compressing the quantized residual signal using statistical characteristics. For example, the entropy coding methods commonly used in H.265 include context-based adaptive binary arithmetic coding (CABAC) and context-based adaptive variable length coding (CAVLC). In the entropy coding step, the encoder selects the most appropriate coding method based on the statistical characteristics of the residual signal and converts it into the final coded bit stream, i.e., the bit stream.

[0277] Based on this, the embodiments of the present application can optimize at least one of the steps in the encoding process: dividing the image blocks, determining the reference frame of the current frame, determining the matching block of the current block, and determining the predicted MV of the current block, based on the intermediate rendering results of the current frame. Optimizing each step can result in a new encoding scheme. Combining the optimization methods of individual steps, the embodiments of the present application provide four corresponding encoding schemes. These four encoding schemes are described in detail below.

[0278] The first encoding scheme

[0279] In video coding standards such as H.265, the encoder typically maintains a decoded picture buffer (DPB), which stores multiple decoded images as reference frames. When encoding the current frame, the encoder selects the most appropriate reference frame from the DPB based on a specific strategy (such as a rate-distortion cost criterion) for motion estimation and compensation. Once a reference frame is selected, the encoder finds the corresponding image block from the reference frame based on the video (MV) and uses it as the predicted value for the current image block. The difference between the predicted value and the actual value (i.e., the prediction residual) is further encoded and transmitted.

[0280] It can be seen that the similarity between the reference frame and the current frame will affect the code stream size of the encoded current frame. That is, the more similar the reference frame is to the current frame, the less residual information there is, the less data needs to be encoded, and the smaller the encoding code stream. Conversely, the larger the encoding code stream. Therefore, selecting an accurate reference frame helps improve encoding efficiency and reduce the amount of data that needs to be encoded.

[0281] In some embodiments, the implementation process of encoding a rendered image based on an intermediate rendering result includes at least the following steps 11 and 12. Steps 11 and 12 are described with respect to the motion compensation step in the inter-frame prediction process, specifically, the optimization scheme for determining the reference image for the current frame. The implementation methods of other steps in the inter-frame prediction process are not limited.

[0282] Step 11: Generate a virtual reference image based on the intermediate rendering results.

[0283] The virtual reference image is a reference frame of the current frame and can be used in the encoding process of the rendered image of the current frame.

[0284] It should be noted that in the embodiments of the present application, the rendered image of the reference frame determined based on the decoded image is referred to as the original reference image. Different from the original reference image, the embodiments of the present application can generate a reference frame image that is more similar to the current frame based on prior information (referred to as a virtual reference image in the embodiments of the present application). This virtual reference image can be used as supplementary information for inter-frame coding to further remove temporal redundancy in the video.

[0285] The prior information includes decoded information inside the encoder (ie, the original reference image, such as the historical frame before the current frame), and information outside the video to be encoded (such as the intermediate rendering results generated during the rendering process).

[0286] In a possible implementation, the implementation process of step 11 may be: based on the rendered image of the historical frame and the CGMV of the current frame, the rendered image of the historical frame is reprojected to generate a virtual reference image corresponding to the current frame.

[0287] After reprojecting the rendered image of the historical frame, the reprojected image may be subjected to validity verification, weighted averaging, hole filling, or intra-frame noise reduction, etc., which is not limited in this embodiment of the present application.

[0288] Among them, reprojection refers to the calculation process of finding the pixel point corresponding to a certain pixel in the current frame based on CGMV.

[0289] It should be noted that, for the specific implementation method of generating the virtual reference frame, reference can also be made to the relevant existing technology, and the embodiments of the present application do not make specific limitations here.

[0290] Step 12: Based on the virtual reference image, encode the rendered image into the bitstream.

[0291] In one possible implementation, the implementation process of the above step 12 may be: based on the virtual reference image, predicting the rendered current frame to obtain a predicted image of the current frame; and encoding the residual between the predicted image and the rendered image according to a relevant coding algorithm.

[0292] In another possible implementation, the implementation process of step 12 may be: based on the virtual reference image, encoding the residual between the virtual reference image and the rendered image of the current frame into the bitstream.

[0293] That is, when a virtual reference image is generated based on the intermediate rendering result, the rendered image of the current frame can be directly encoded using the virtual reference image; or the virtual reference image can be used to determine a predicted image for encoding the current frame, and then the predicted image can be used to encode the rendered image of the current frame. This embodiment of the present application does not impose any restrictions on this, and can be flexibly selected based on actual needs and business scenarios.

[0294] It should be understood that when encoding the rendered image in a block-by-block manner, the virtual reference image in the embodiment of the present application can also be replaced by a virtual reference block corresponding to an image block in the current frame, that is, the virtual reference image can be replaced by a virtual reference block.

[0295] In summary, when determining the reference frame for the current frame, the target server can generate a virtual reference image based on the intermediate rendering results, and then encode the rendered image of the current frame based on the virtual reference image. Because the virtual reference image is more similar to the rendered image of the current frame, the virtual reference image can provide a more accurate reference for encoding the rendered image of the current frame, thereby improving inter-frame prediction accuracy, thereby improving encoding efficiency and reducing the bit rate when encoding the rendered image of the current frame.

[0296] The second encoding scheme

[0297] As previously explained, compared to directly encoding the current frame, segmenting the rendered image of the current frame into smaller regions, allowing each region to be encoded separately, can improve coding efficiency and compression performance to a certain extent. Therefore, the segmentation method and result of segmenting the rendered image of the current frame are both related to coding efficiency and compression performance.

[0298] In HEVC (H.265), the process of dividing a CTU into smaller CUs is primarily achieved through quadtree (QT) partitioning. However, QT partitioning only allows for binary partitioning along either the horizontal or vertical direction, dividing a larger image block into four smaller sub-blocks. This partitioning method is relatively simple, but may not achieve optimal coding performance when processing images with complex textures or motion characteristics. To overcome the limitations of the block partitioning algorithm in H.265, VCC (H.266) introduces binary tree (BT) and ternary tree (TT) partitioning. BT partitioning allows for asymmetric binary partitioning of image blocks along either the horizontal or vertical direction, while TT partitioning allows for tri-partitioning of image blocks along either the horizontal or vertical direction. Furthermore, VVC introduces the geometric partitioning mode (GPM). Unlike traditional QT, BT, and TT partitioning methods, GPM does not simply partition blocks along the horizontal or vertical direction, but instead divides blocks based on the actual motion trajectory and shape of the object.

[0299] Based on this, when the embodiment of the present application uses GPM to divide the rendered image of the current frame into blocks, it can be combined with the depth map of the current frame to assist in determining the division information of each image block in the current frame.

[0300] Taking GPM in H.266 as an example, when the intermediate rendering result of the current frame includes the depth map of the current frame, the process of encoding the rendered image based on the intermediate rendering result in the embodiment of the present application includes at least the following steps 21-22. Among them, steps 21-22 are for the block segmentation step in the inter-frame prediction process, specifically an optimization scheme for determining the segmentation line of the image block based on the depth map, and do not limit the implementation methods of other steps in the inter-frame prediction process.

[0301] Step 21: Based on the depth map, generate partition information of a first image block, where the first image block is an image block of a rendered image. The partition information is used to partition the first image block to obtain a plurality of partitions.

[0302] For the current frame, mesh data of the current frame may be used for rasterization to generate a depth map of the current frame. Each pixel value in the depth map represents the distance from the position to the camera (also called depth).

[0303] Taking GPM as an example, the first image block may be any CU in the current frame rendering image, and the division information of the first image block may be the position of a dividing line that divides the first image block into two.

[0304] In one possible implementation, the implementation process of step 21 may be: based on the depth map of the current frame, determining a second partitioning information set from a first partitioning information set, the first partitioning information set including multiple partitioning information, the second partitioning information set being a subset of the first partitioning set; determining the partitioning information of the first image block from the second partitioning set.

[0305] The first division information set includes 64 division modes defined by GPM, that is, 64 dividing lines; the second division information set includes at least one of the 64 division modes.

[0306] It should be noted that GPM uses a geometrically positioned dividing line to divide the CU into two parts. The position of this dividing line is based on the angle parameter of the specific partition (denoted as ) and the offset parameter (denoted as ρ). The GPM in H.266 stipulates that 360° is quantized into 24 different And each There are at most 4 kinds of ρ, which can produce 64 partitioning modes in total. Each partitioning mode corresponds to a and a ρ. When determining the partition line corresponding to a CU, rate-distortion optimization (RDO) is usually used to select the partition mode with the lowest rate-distortion cost from the encoding results under 64 partition modes as the partition mode for dividing the CU into blocks. In other words, for a CU, GPM needs to calculate the rate-distortion cost under 64 modes and determine the partition mode with the lowest rate-distortion cost as the partition mode for the CU. However, the RDO operation in the above process of determining the CU partition mode is very time-consuming, increasing the complexity of the overall calculation.

[0307] Based on this, the embodiment of the present application uses the depth map of the current frame to filter out the 64 division modes to determine the screening range of the dividing line of each CU in the current frame, so that only a small number of division modes need to be calculated for the rate-distortion cost, which greatly reduces the data calculation amount of the RDO operation and reduces the computational complexity.

[0308] In one possible implementation, the implementation process of determining the second segmentation information set from the first segmentation information set based on the depth map is: based on the depth map, determining the position of the depth change in the first image block; based on the position of the depth change, determining the second segmentation information set from the first segmentation information set.

[0309] The position where the depth changes in the first image block may be a contour line between the foreground object and the background area in the first image block.

[0310] In one possible implementation, the contour line in the first image block is analyzed to determine its direction and angle. According to the angle of the contour line, several division modes close to the position of the contour line are selected from the 64 division modes specified by GPM. and ρ, these screened The combination of and ρ is the second partition information set.

[0311] Optionally, considering that the contour lines of image blocks may not be clear enough or may contain noise in certain complex scenes, after determining the contour line of the first image block, the contour line may be optimized, such as by denoising, to improve the accuracy of the contour line of the first image block. This embodiment of the present application is not limited to this.

[0312] Step 22: Encode the partition information into the bitstream.

[0313] In order to facilitate the decoding end to accurately reconstruct the first image block, when the first image block is divided based on the division information, the division information also needs to be encoded into the bitstream when encoding the first image block.

[0314] In summary, when the target server uses GPM to divide the image blocks in the current frame into smaller areas, it can use the depth map of the current frame to narrow the search range of the segmentation line, so that only those areas that match the filtered areas are considered when determining the segmentation pattern. The partitioning pattern corresponding to ρ. In this way, the 64 partitioning patterns that originally need to be traversed can be reduced to a smaller set, thereby significantly reducing the amount of GPM calculations, reducing computational complexity, and improving coding efficiency.

[0315] It should be noted that the second coding scheme mentioned above is only exemplified by H.266 and GPM, and does not constitute a limitation on the applicable scenarios and objects of the above coding scheme. With the development of coding and decoding technology, the image block of the current frame may not be a CU, but a sub-block or area determined based on other division methods; the above GPM defined and ρ may also evolve into other forms of parameters / division methods, but the inventive concept of determining the segmentation line of an image block based on a depth map provided in the embodiment of the present application is also applicable to similar problems and scenarios.

[0316] Based on the above-mentioned first encoding scheme and second encoding scheme, when encoding the rendered image in units of image blocks, if the rendering intermediate result includes the graphics motion vector (CGMV) of the current frame; then encoding the rendered image based on the rendering intermediate result includes: determining the MV of the second image block based on the graphics motion vector, and encoding the MV of the second image block; the second image block is any image block in the rendered image of the current frame.

[0317] It should be understood that the second image block here can be the same image block as the first image block in step 21 above, or a different image block, and this embodiment of the present application does not limit this. The second image block can be an image block obtained by traditional division or an image block obtained by the second encoding scheme described above. This application does not limit the division method of the second image block.

[0318] It should be noted that the MV of the second image block here is a real MV, which is different from the predicted MV below. That is, for the second image block, there is a real MV and a predicted MV. The predicted MV can be used in the encoding process of the real MV to reduce the amount of encoded data.

[0319] The third encoding scheme described below illustrates the implementation process of determining the MV of the second image block, and the fourth encoding scheme described below illustrates the implementation process of encoding the MV of the second image block. The third and fourth encoding schemes can be independently applied to the MV encoding process of the second image block, or can be used in combination to encode the MV of the second image block, and this application does not impose any restrictions on this.

[0320] The third encoding scheme

[0321] As previously explained, the motion estimation phase of inter-frame prediction is a key step in image compression, significantly improving compression efficiency by eliminating temporal redundancy. The TZSearch algorithm (a computational estimation algorithm) is a fast search algorithm within this phase. This algorithm employs a fast search strategy designed to significantly reduce search time while ensuring encoding quality. It utilizes a hybrid search model and designs a variety of search templates, such as square, diamond, and grating, to address the varying motion states of objects in video / image sequences. Furthermore, the TZSearch algorithm no longer uses the origin as the search starting point. Instead, it comprehensively considers the characteristics of the video / image sequence and employs advanced motion vector prediction (AMVP) technology to predict a new starting search point.

[0322] Taking the TZSearch algorithm as an example, when the intermediate rendering result of the current frame includes the CGMV of the current frame, the process of determining the MV of the second image block based on the CGMV in the embodiment of the present application includes at least the following steps 31-34. Among them, steps 31-34 are for the motion estimation link in the inter-frame prediction process, specifically explaining the optimization scheme for searching for the matching block corresponding to the second image block based on the CGMV, and do not limit the implementation methods of other links in the inter-frame prediction process.

[0323] Step 31: Based on the CGMV of the current frame, determine the predicted MV of the key pixel points in the second image block, where the key pixel points include one or more pixel points.

[0324] It should be noted that the key pixel point in the second image block can be the central pixel point of the second image block, or the four corner points of the second image block, or can include other key points. The embodiment of the present application does not limit the number of key pixel points.

[0325] Among them, for the current frame, the target server can generate a motion vector based on the depth map in the intermediate result of the current frame rendering, and the camera matrix (including the intrinsic and extrinsic parameters of the camera) using computer graphics methods to obtain the CGMV of the current frame. Of course, the CGMV of the current frame can also be obtained directly from the rendering pipeline of the graphics rendering engine. The embodiment of the present application does not limit the method of obtaining the CGMV. Based on this, when the CGMV of the current frame is determined, the CGMV of each pixel in the current frame is determined.

[0326] In a possible implementation, for a key pixel in the second image block, the CGMV of the key pixel is directly determined as the predicted MV of the key pixel. That is, the predicted MV of the key pixel is the CGMV.

[0327] Of course, when determining the predicted MV of the key pixel point, the CGMV of the key pixel point can also be used as prior information to predict the MV of the key pixel point in the second image block. The prediction can be based on the CGMV of the adjacent pixel points for interpolation or averaging to obtain the predicted MV of the key pixel point in the current image block.

[0328] Step 32: Determine the search range based on the predicted MV of the key pixel point.

[0329] In a possible implementation, the implementation process of step 32 may be: determining the motion trend of the second image block within a search window according to the predicted MV of the key pixel point; and determining the search range within the search window based on the motion trend.

[0330] The search window is an area in the rendered image of the reference frame for searching for a matching block of the second image block. The search window includes multiple image blocks. The image block located in the center area of ​​the search window corresponds to the position area of ​​the second image block in the rendered image of the current frame.

[0331] It should be understood that the size of the search range determined by the predicted MV of the key pixel point is smaller than the size of the search window.

[0332] Step 33: Determine a matching block corresponding to the second image block from a reference rendered image according to the search range, where the reference rendered image is a rendered image of a reference frame corresponding to the current frame.

[0333] It should be noted that the reference rendered image is the rendered image of the reference frame corresponding to the current frame. The reference rendered image can be a reference image determined in a traditional manner, that is, an original reference image, or a virtual reference image determined using the first encoding scheme described above. This embodiment of the present application does not impose any restrictions on this.

[0334] As an example, referring to Figure 8, for a second image block in the rendered image of the current frame, a search window for searching for a matching block can be determined from the rendered image of the reference frame (i.e., the reference rendered image) based on its position in the rendered image. Taking a square search template as an example, which defines 25 search points, the TZSearch algorithm calculates the rate-distortion cost of these 25 pixels within the search window to determine the matching block corresponding to the second image block.

[0335] In an embodiment of the present application, based on the predicted MV of the key pixel points of the second image block, initial search points are determined from a square search template, such as search points a, b, c, and d shown in Figure 8. Further, based on the initial search points a, b, c, and d, a search range is determined, which includes multiple candidate search points closest to the initial search points (search points surrounded by dotted boxes in Figure 8). For these candidate search points, the TZSearch algorithm determines the optimal point by calculating the rate-distortion cost, and then determines the image block to which the optimal point belongs as the matching block corresponding to the second image block.

[0336] The implementation process of determining the matching block corresponding to the second image block based on the search range (or candidate search point) can refer to the TZSearch algorithm, which is not described in detail or limited in the embodiments of the present application.

[0337] Step 34: Determine the MV of the second image block based on the second image block and the matching block corresponding to the second image block.

[0338] The MV of the second image block can be determined based on the motion offset of the second image block relative to the matching block, which will not be elaborated here.

[0339] In summary, the target server estimates the motion trend of the second image block based on the predicted MV of the key pixels in the second image block, narrowing the TZSearch algorithm's search range and focusing it more closely on the possible best-matching region. Because the search range is narrowed, the TZSearch algorithm only needs to calculate the rate-distortion penalty for a small number of pixels, significantly reducing the amount of data required for the RDO operation and lowering its computational complexity.

[0340] The fourth encoding scheme

[0341] As previously explained, in the inter-frame prediction process, the image block of the current frame searches for the best matching block in the encoded reference frame. This process is called motion estimation, and the displacement between the matching block and the current block is the MV. To facilitate the decoder's search for the corresponding matching block in the reference image when reconstructing the image block, the MV of the image block must also be encoded in the bitstream during encoding.

[0342] In practical applications, the MV of the current image block is usually not directly encoded into the bitstream. Instead, the residual between the MV of the current image block and the predicted MV is encoded into the bitstream. The predicted MV is determined using MV prediction technology. MV prediction uses adjacent blocks or spatiotemporal correlation to predict the MV of the current block. For example, the AMVP and Merge modes in H.265 use this prediction technology to encode the MV of image blocks.

[0343] AMVP uses the MVs of adjacent image blocks to predict the MV of the current image block and encodes the prediction residual (i.e., the difference between the actual MV and the predicted MV). In Merge mode, the MV of the current image block is not directly encoded and transmitted. Instead, the optimal MV is selected from the set of MVs of adjacent encoded image blocks as the MV of the current image block.

[0344] When the intermediate rendering result of the current frame includes the CGMV of the current frame, the process of encoding the MV of the second image block in the embodiment of the present application includes at least the following steps 41-43. Among them, steps 41-43 are for the MV prediction link in the inter-frame prediction process, specifically for explaining the optimization scheme for encoding the MV of the second image block, and do not limit the implementation methods of other links in the inter-frame prediction process.

[0345] Step 41: Based on the CGMV of the current frame, determine the predicted MV of the key pixel points in the second image block, where the key pixel points include one or more pixel points.

[0346] The implementation process of step 41 may refer to the above-mentioned step 31, and the embodiment of the present application does not limit this.

[0347] As an example, the key pixel points may be the four corner points of the second image block.

[0348] Step 42: Determine the predicted MV of the second image block based on the predicted MV of the key pixel point.

[0349] In a possible implementation, the implementation process of step 42 may be: constructing a candidate MV list of the second image block, the candidate MV list including predicted MVs of key pixels in the second image block; and determining the predicted MV of the second image block based on the candidate MV list.

[0350] The candidate MV list includes multiple candidate MVs, and the multiple candidate MVs include predicted MVs of key pixel points.

[0351] Taking H.265 as an example, the candidate MV list of the second image block can be a candidate MV list of AMVP or a candidate MV list of Merge mode, and the embodiment of the present application does not limit this.

[0352] In AMVP, a candidate MV list is constructed containing multiple MV candidates, each of which is derived from the MV of an adjacent image block of the second image block. When constructing the candidate MV list, factors such as the position and motion characteristics of adjacent image blocks are considered to ensure that the MV candidates in the candidate MV list have similar motion characteristics to the second image block.

[0353] In Merge mode, for the second image block, a candidate MV list is constructed based on its spatially adjacent (e.g., left, top, top right, bottom left, etc.) and / or temporally adjacent (e.g., image blocks at corresponding positions in the previous frame) coded image blocks. This candidate MV list typically contains multiple candidate MVs, each of which is selected from the MVs of adjacent image blocks.

[0354] In an embodiment of the present application, after determining the predicted MV of the key pixel point in the second image block, the predicted MV of the key pixel point is also added to the candidate MV list of the second image block, that is, the predicted MV of the key pixel point will participate in the calculation process of determining the predicted MV of the second image block.

[0355] For each candidate MV in the candidate MV list, a similarity measure (e.g., rate-distortion cost) is calculated between each candidate MV and the (actual) MV of the second image block, so as to select a candidate MV from the candidate MV list that best matches the motion of the second image block and determine it as the predicted MV of the second image block.

[0356] It should be noted that, when the predicted MVs of the key pixels of the second image block are candidate MVs in the candidate MV list, the predicted MVs of the second image block determined by the RDO operation may also be the predicted MVs of these key pixels. This embodiment of the present application does not impose any limitation on this.

[0357] Step 43: Encode the MV of the second image block based on the predicted MV of the second image block.

[0358] Taking H.265 as an example, the methods of encoding the MV of the image block include AMVP and Merge mode. The following introduces the implementation process of encoding the second image block in these two ways.

[0359] In one possible implementation, the implementation process of encoding the MV of the second image block using AMVP may be: obtaining an MV residual based on the MV of the second image block and the predicted MV of the second image block; encoding the MV residual into the bitstream, and encoding the index information of the predicted MV in the candidate MV list into the bitstream; the index information indicates the position of the predicted MV of the second image block in the candidate MV list.

[0360] Since the MVs of adjacent image blocks and the predicted MVs of key pixels are similar to the MV of the second image block, the prediction residuals are usually small. Encoding the MV residuals into the bitstream can improve coding efficiency and reduce the number of bits required for coding.

[0361] In one possible implementation, the implementation process of encoding the MV of the second image block in the Merge mode can be: encoding the index information of the predicted MV in the candidate MV list into the bitstream to achieve the encoding of the MV of the second image block; the index information indicates the position of the predicted MV of the second image block in the candidate MV list.

[0362] Since the length of the candidate MV list of AMVP in H.265 is usually short (usually, the length of the candidate list of AMVP is 2), the number of bits required for encoding the index information is relatively small, which can save the encoding bit rate.

[0363] As can be seen, when encoding the MV of the second image block using AMVP, the residual between the predicted MV of the second image block and the MV of the second image block, as well as the index information of the predicted MV in the candidate MV list, are encoded into the bitstream. When encoding the MV of the second image block in Merge mode, only the index information of the predicted MV in the candidate MV list is encoded into the bitstream. The two methods encode different information into the bitstream. Compared with AMVP, the amount of data used to encode the MV of the second image block in Merge mode is smaller, further reducing the encoding bitrate.

[0364] It should be noted that there are multiple encoding methods for encoding the MV of the second image block. In practical applications, a suitable encoding method can be selected based on specific encoding requirements, computational complexity, and coding efficiency, including but not limited to the AMVP and Merge modes exemplified in the above embodiments.

[0365] To sum up, when the target server encodes the MV of the second image, it adds the predicted MV of the key pixel points in the second image block to the candidate MV list, thereby improving the similarity between the predicted MV and the MV of the second image block, thereby reducing the encoding bit rate of the MV of the second image block encoded in AMVP and Merge modes, and improving the encoding efficiency.

[0366] In an embodiment of the present application, scene data is transmitted via a code stream, so that the decoding end (or the end receiving the code stream) does not need to pre-download and store the scene data required to run the three-dimensional virtual scene. At the same time, when transmitting the scene data of the current frame, the scene data of the encoding end and the decoding end can remain consistent. Based on this, the encoding end can effectively utilize the scene data in the process of encoding the rendered image of the current frame, and optimize the encoding process of the rendered image to reduce the bit rate and computational overhead of the encoded rendered image. Furthermore, when using scene data to encode the rendered image can reduce the bit rate, this encoding method can also reduce the bandwidth resources for transmitting the code stream between the encoding end and the decoding end.

[0367] In addition, for the game scene, the encoding end of the encoding method provided by the present application can be a cloud platform, and the corresponding decoding end can be a client, that is, the cloud platform and the client use an end-cloud collaborative approach to run the three-dimensional virtual scene. After the cloud platform determines the scene data of the current frame, it can encode the scene data into a code stream and send it to the client. In this way, the client does not need to pre-download and store the scene data required to run the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience. In the case where the cloud platform and the client have the same scene data, the cloud platform can effectively use the scene data in the process of encoding the rendered image of the current frame and optimize the encoding process of the rendered image; correspondingly, the client can decode the code stream based on the same scene data and process the decoding information in a corresponding manner to obtain the rendered image reconstructed by the current frame. In this way, not only can the bit rate and computational overhead of the cloud platform's encoded rendered image be reduced, but the client's own processing power can also be effectively utilized, thereby reducing the bandwidth resources for transmitting the code stream between the cloud platform and the client.

[0368] FIG9 is a flowchart of a decoding method provided in an embodiment of the present application. Taking the decoding method applied to a client as an example, please refer to FIG9 . The method includes the following steps.

[0369] Step 901: Obtain scene data of the current frame in the three-dimensional virtual scene from the code stream.

[0370] In some embodiments, the scene data includes local scene data and / or scene change data, wherein the local scene data refers to scene data newly added to the current frame relative to the historical frame, and the scene change data refers to scene data whose state has changed relative to the historical frame.

[0371] For the relevant description of the scene data of the current frame, please refer to the relevant description in the previous encoding method, which will not be repeated here.

[0372] In some embodiments, when the decoding method is applied to the client, before executing the above step 901, the decoding method also includes: sending scene interaction information to the target server, the scene interaction information indicating the operation triggered by the operator in the three-dimensional virtual scene, and the position of the operator in the three-dimensional virtual scene.

[0373] For instructions on the cloud platform and target server, please refer to the relevant instructions in the encoding method above, which will not be repeated here.

[0374] In some embodiments, when the decoding method is applied to the client, the client can also obtain the operating mode of the three-dimensional virtual scene; if the operating mode is end-cloud collaborative operation, the above-mentioned step of sending scene interaction information to the target server in the cloud platform is executed.

[0375] That is, when the client interacts with the cloud platform, the client operator can select the operation mode of the three-dimensional virtual scene in the client, and when the selected operation mode is end-cloud collaborative operation, the client sends scene interaction information to the target server, so that the target server determines the scene data of the current frame based on the scene interaction information, thereby executing the encoding method shown in Figure 7 above.

[0376] In one possible implementation, the operator can select either cloud or client-side operation mode for the 3D virtual scene. In cloud operation mode, the target server independently runs the 3D virtual scene and sends rendered images to the client via video streaming. In client-side operation mode, the client runs the 3D virtual scene locally, renders the scene data to generate rendered images, and displays the rendered images without interacting with the cloud platform.

[0377] Step 902: Decode the code stream based on the scene data to obtain a rendered image reconstructed for the current frame.

[0378] In one possible implementation, the implementation process of step 902 may be: obtaining an intermediate rendering result of the current frame, where the intermediate rendering result is obtained by rendering scene data; decoding the code stream based on the intermediate rendering result to obtain a rendered image reconstructed for the current frame.

[0379] As an example, after obtaining the scene data, the scene data can be input into a graphics rendering engine, and the graphics rendering engine can be used to render the scene data of the current frame to obtain an intermediate rendering result and a low-quality rendered image (optional, and no rendered image may be obtained).

[0380] It should be understood that, when the cloud platform and the client both have the same scene data, the intermediate rendering result obtained by the client rendering the scene data of the current frame is the same as the intermediate rendering result generated by the target server in the cloud platform. Since the computing power of the client may be lower than that of the target server, the image quality of the rendered image rendered by the graphics rendering engine of the client (wherein, image quality may include subjective quality and objective quality (objective quality such as rate-distortion performance, peak signal to noise ratio (PSNR)), etc.) may be lower than the image quality of the rendered image rendered by the graphics rendering engine of the target server, or even the client may not be able to render the rendered image. Therefore, in the embodiment of the present application, the client only decodes the received code stream, and uses the intermediate rendering result to assist in reconstructing the rendered image of the current frame.

[0381] In some embodiments, based on the scene data of the current frame, the implementation process of obtaining the rendering intermediate result of the current frame can be: reconstructing the scene based on the scene data of the current frame to obtain the reconstructed scene of the current frame; rendering the reconstructed scene of the current frame to obtain the rendering intermediate result of the current frame.

[0382] In one possible implementation, when the scene data includes local scene data and scene change data, scene reconstruction is performed based on the local scene data and scene data of historical frames to obtain a reconstructed scene for the current frame; the reconstructed scene for the current frame is updated based on the scene change data; and based on the updated reconstructed scene for the current frame, an intermediate rendering result for the current frame is obtained. The intermediate rendering result is obtained by rendering the updated reconstructed scene for the current frame by a graphics rendering engine.

[0383] As an example, the historical frame may be a previous frame adjacent to the current frame, or multiple frames that are continuous in time sequence. The embodiment of the present application does not limit the number of historical frames.

[0384] In another possible implementation, when the scene data includes local scene data, scene reconstruction is performed based on the local scene data and scene data of historical frames to obtain a reconstructed scene for the current frame; and based on the reconstructed scene for the current frame, an intermediate rendering result for the current frame is obtained. The intermediate rendering result here is obtained by rendering the reconstructed scene for the current frame by a graphics rendering engine.

[0385] That is, when the scene data of the current frame overlaps with the scene data of the historical frame, and the scene data of the historical frame has been sent to the client, the client can reuse the scene data of the historical frame when reconstructing the scene, so as to combine the scene data of the historical frame and the newly added scene data of the current frame to render and generate the reconstructed scene of the current frame.

[0386] It should be understood that as the amount of scene data synchronized between the cloud platform and the client increases, in the subsequent process of transmitting rendered images, the cloud platform and the client need to synchronize less and less new scene data, or even no new scene data. At this time, the client independently determines the scene data of the current frame based on the scene data of the historical frames it has received and the operator's scene interaction information, and renders it to generate a virtual reference image.

[0387] In another possible implementation, when the scene data includes scene change data, the scene is reconstructed based on the scene data of the historical frame to obtain a reconstructed scene for the current frame; the reconstructed scene for the current frame is updated based on the scene change data; and based on the updated reconstructed scene for the current frame, an intermediate rendering result for the current frame is obtained. The intermediate rendering result is obtained by rendering the updated reconstructed scene for the current frame by a graphics rendering engine.

[0388] It should be understood that whether the scene data includes scene change data depends on the operating logic of the three-dimensional virtual scene and the operations triggered by the operator in the three-dimensional virtual scene. If the operator does not trigger any operation, there may be no scene change data in the current frame. At this time, there is no need to transmit scene change data between the cloud platform and the client.

[0389] It should be noted that the client reconstructs the scene based on the scene data of the current frame, generates the reconstructed scene of the current frame, and renders the reconstructed scene to obtain the specific implementation method of the rendering intermediate result of the current frame. Please refer to the relevant technology. The embodiment of this application does not limit this, so it will not be repeated here.

[0390] In some embodiments, when the information carried in the bitstream is different, the implementation process of decoding the bitstream based on the intermediate rendering result of the current frame to obtain the rendered image reconstructed for the current frame will also be different. Next, for the encoding scheme shown in the embodiment of Figure 7, the present application provides two corresponding decoding schemes, which are described below.

[0391] The first decoding scheme

[0392] For the first encoding scheme described above, i.e., using a virtual reference image to encode the rendered image of the current frame into the bitstream, embodiments of the present application provide a corresponding decoding scheme. After obtaining the intermediate rendering result of the current frame, the virtual reference image can be generated using the intermediate rendering result, and the rendered image of the current frame can be reconstructed using the virtual reference image.

[0393] In some embodiments, the implementation process of decoding the code stream based on the intermediate rendering result of the current frame to obtain the rendered image reconstructed for the current frame includes at least the following steps 51-52.

[0394] Step 51: Generate a virtual reference image based on the intermediate rendering result of the current frame.

[0395] It should be understood that after the client decodes and obtains the scene data of the current frame, the target server and the client can obtain the same rendering intermediate result after rendering the same scene data of the current frame; and based on the same rendering intermediate result, the client can also use a similar method to generate the same virtual reference image as the target server.

[0396] In other words, for the current frame, the client executes step 51 in a manner that is the same / similar to the way in which the cloud platform generates a virtual reference image in the embodiment shown in FIG7 above. Therefore, for the relevant explanation of step 51, please refer to the explanation of step 11 in the first encoding scheme above, and no further details will be given here.

[0397] Step 52: Decode the code stream based on the virtual reference image to obtain a rendered image reconstructed for the current frame.

[0398] As described above, when the residual between the virtual reference image and the rendered image of the current frame is encoded in the bitstream, the client can obtain the residual of the current frame after decoding the bitstream, and thus use the virtual reference image of the current frame and the residual to determine the rendered image reconstructed for the current frame.

[0399] The second decoding scheme

[0400] For the fourth encoding scheme described above, i.e., a scheme in which the (actual) MV of a key pixel in an image block is encoded into a bitstream using the predicted MV of the key pixel in the image block, an embodiment of the present application provides a corresponding decoding scheme. After obtaining the intermediate rendering result of the current frame, the intermediate rendering result can be used to determine the predicted MV of the key pixel of the second image block in the current frame, and the predicted MV of the key pixel of the second image block can be used to determine the MV of the second image block, and the MV can be used to find a matching block to reconstruct the rendered image of the current frame.

[0401] In some embodiments, taking the reconstructed image block as an example, when the rendering intermediate result of the current frame includes the CGMV of the current frame, the code stream is decoded based on the rendering intermediate result of the current frame to obtain the rendered image reconstructed by the current frame. The implementation process is: decoding the code stream based on the CGMV to obtain the motion vector MV of the second image block, and the second image block is any image block in the rendered image.

[0402] Furthermore, the implementation process of decoding the code stream based on CGMV to obtain the motion vector MV of the second image block at least includes the following steps 61-63.

[0403] Step 61: Based on the CGMV of the current frame, determine the predicted MV of key pixel points in the second image block, where the key pixel points include one or more pixel points.

[0404] Among them, the implementation process of determining the predicted MV of the key pixel points in any image block in the current frame based on CGMV can refer to the description of step 31 or step 41 in the embodiment of Figure 7. The implementation logic of the two is similar, so it will not be repeated here.

[0405] Step 62: Construct a candidate MV list of the second image block, where the candidate MV list includes predicted MVs of key pixels.

[0406] The implementation process of constructing the candidate MV list of the second image block can refer to the description of step 42 in the embodiment of FIG. 7 . The implementation logic of the two is similar, so it will not be described in detail here.

[0407] Step 63: Determine the MV of the second image block based on the candidate MV list.

[0408] Regarding the second image block, as described in the fourth encoding scheme above, when encoding the MV of the second image block, the Merge mode can be used to encode index information indicating the position of the predicted MV in the candidate MV list into the bitstream. Alternatively, the AMVP method can be used to encode the index information and the MV residual corresponding to the second image block into the bitstream. Correspondingly, when decoding and determining the MV of the second image block, there are two corresponding implementation methods for determining the MV of the second image block.

[0409] In the first implementation method, the code stream carries index information and the MV residual corresponding to the second image block. The index information indicates the position of the MV of the second image block in the candidate MV list. At this time, the implementation process of determining the MV of the second image block can be: obtaining the candidate MV of the second image block from the candidate MV list based on the index information; and obtaining the MV of the second image block based on the candidate MV of the second image block and the MV residual corresponding to the second image block.

[0410] In the second implementation method, the code stream carries index information, which indicates the position of the MV of the second image block in the candidate MV list. At this time, the implementation process of determining the MV of the second image block can be: obtaining the MV of the second image block from the candidate MV list based on the index information, that is, taking the MV indicated by the index information in the candidate MV list as the MV of the second image block.

[0411] That is, regardless of AMVP or Merge mode, after determining the predicted MV of the key pixel point in the second image block, the embodiment of the present application also adds the predicted MV of the key pixel point to the candidate MV list of the second image block, that is, the predicted MV of the key pixel point will participate in the calculation process of determining the predicted MV of the second image block.

[0412] The difference between AMVP and Merge modes is that after determining the candidate MV for the second image block from the candidate MV list (equivalent to the "predicted MV of the second image block" described on the encoding side), AMVP determines the MV of the second image block based on the MV residual parsed in the bitstream and the candidate MV. Merge mode, on the other hand, directly determines the candidate MV as the MV of the second image block.

[0413] In summary, the cloud platform and client in the embodiment of the present application use a terminal-cloud collaborative mode to run a three-dimensional virtual scene. After the cloud platform sends the scene data of the current frame and the rendered image of the current frame encoded using the scene data to the client via a code stream, the client, in combination with its own image rendering capabilities, uses the scene data of the current frame for auxiliary decoding to obtain a rendered image reconstructed by the current frame. In this process, the client does not need to pre-download and store the scene data required to run the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience. Moreover, when the scene data of the cloud platform and the client can be consistent, the cloud platform can effectively utilize the scene data in the process of encoding the rendered image of the current frame and optimize the encoding process of the rendered image to reduce the bit rate and computational overhead of the encoded rendered image. Furthermore, when using scene data to encode the rendered image can reduce the bit rate, this encoding method can also reduce the bandwidth resources for transmitting the code stream between the encoding end and the decoding end. Correspondingly, after parsing and obtaining the scene data of the current frame, the client can decode the code stream based on the same scene data and process the decoding information in a corresponding manner to obtain a rendered image reconstructed by the current frame. This not only reduces the bit rate and computational overhead of encoding and rendering images on the cloud platform, but also effectively utilizes the client's own processing power, thereby reducing the bandwidth resources for transmitting code streams between the cloud platform and the client.

[0414] Regarding the encoding and decoding method provided in the embodiment of the present application, FIG7 above only takes the target server in the cloud platform as an example to explain the specific implementation process of the encoding method; FIG9 above only takes the client as an example to explain the specific implementation process of the decoding method. Therefore, in the case where the target server in the cloud platform communicates directly with the client, the embodiment shown in FIG7 and FIG9 above can be combined to adopt the end-cloud collaborative mode to execute the encoding and decoding method of the embodiment of the present application, and the end-cloud collaboratively runs the three-dimensional virtual scene, and both the target server and the client perform scene data rendering and generate a virtual reference image.

[0415] In some embodiments, when the cloud platform includes multiple servers and the multiple servers are deployed in a distributed manner, the encoding method shown in Figure 7 can be performed by a central server in the cloud platform. In this case, the central server can send the encoded bitstream directly to the client or to another edge server, such as the target server with the closest communication distance to the client.

[0416] Based on this, the execution entity of the decoding method provided in the embodiment of the present application can be a target server or a client, and the embodiment of the present application does not limit this.

[0417] When the target server executes the decoding method and obtains the rendered image reconstructed from the current frame, it can continue to use the method of the embodiment of the present application, use a virtual reference image to encode the reconstructed rendered image, and send it to the client; it can also use the traditional video streaming method, use the reconstructed image of the historical frame, encode the reconstructed image of the current frame, and send it to the client. The embodiment of the present application does not impose any restrictions on this.

[0418] In some embodiments, when the cloud platform includes multiple servers and the multiple servers are deployed in a distributed manner, the central server can send the data of the three-dimensional virtual scene to the target server so that the target server stores the data of the three-dimensional virtual scene and executes the encoding method shown in Figure 7 above based on the data of the three-dimensional virtual scene.

[0419] Based on this, the decoding method provided in the embodiment of the present application can be executed by the client, that is, the client directly interacts with the target server and operates the three-dimensional virtual scene in a terminal-cloud collaborative mode.

[0420] In some embodiments, when the local area network where the client is located includes a local server, the central server or target server in the cloud platform can send the data of the three-dimensional virtual scene to the local server, so that the local server stores the data of the three-dimensional virtual scene and executes the encoding method shown in Figure 7 above based on the data of the three-dimensional virtual scene.

[0421] Similarly, when the local server executes the decoding method and obtains the reconstructed image of the current frame, it can continue to use the method of the embodiment of the present application, use the virtual reference image to encode the reconstructed rendering image, and send it to the client; it can also use the traditional video streaming method, use the reconstructed image of the historical frame, encode the reconstructed image of the current frame, and send it to the client. The embodiment of the present application does not impose any restrictions on this.

[0422] To sum up, the encoding and decoding method provided in the embodiment of the present application can be implemented in an end-cloud collaboration (i.e., central server-client, or target server-client) manner, or in an end-edge-cloud collaboration (i.e., central server-target server-client, or target server-local server-client, or central server-local server-client) mode. The embodiment of the present application does not impose any restrictions on this. It can be flexibly set according to the image / video to be encoded and decoded, or flexibly set based on the encoding and decoding scenario. The embodiment of the present application does not impose any restrictions on this.

[0423] Next, with reference to Figures 10 to 13 , the specific process of executing the encoding and decoding method shown in the embodiment of the present application in end-cloud collaboration, end-edge collaboration, and end-edge-cloud collaboration is exemplified.

[0424] It should be noted that, for the cloud platform or server that performs the encoding operation, the following only uses the virtual reference image to assist in encoding the current frame as an example, but in actual applications, the process of encoding the current frame can also be combined with other encoding schemes other than the first encoding scheme in the embodiment of Figure 7 above, such as the second encoding scheme to the fourth encoding scheme. Similarly, for the client that performs the decoding operation, the following also only uses the virtual reference image to assist in decoding the current frame as an example, but in actual applications, the encoding scheme adopted by the reference encoding side and the process of decoding and reconstructing the current frame can also refer to other decoding operations in the embodiment of Figure 8 above. The embodiments of the present application do not limit this, and will not be repeated here one by one.

[0425] Refer to Figure 10, which is a schematic diagram of an end-cloud collaborative encoding and decoding process provided by an embodiment of the present application. For the server in the cloud platform, when executing the encoding method provided by an embodiment of the present application, the camera posture synchronization is performed based on the scene interaction information of the operator sent by the client, and the scene data of the current frame is determined based on the position of the virtual camera, and the scene data of the current frame is rendered to obtain the rendered image of the current frame and the rendering intermediate result, and then a virtual reference image is generated based on the rendering intermediate result. During encoding transmission, the scene data of the current frame is encoded into the bitstream, and the rendered image of the current frame is auxiliary encoded based on the virtual reference image to encode the rendered image of the current frame into the bitstream, and then the encoded bitstream is sent to the client through the network.

[0426] The server may be any server in the cloud platform that can communicate with the client, such as a target server that is closest to the client.

[0427] After receiving the code stream from the server on the cloud platform, the client decodes the code stream to obtain the scene data for the current frame, such as local scene data and / or scene change data for the current frame. The client then reconstructs the scene based on the scene data for the current frame to obtain a reconstructed scene for the current frame. If scene change data is decoded, the reconstructed scene is updated using this scene change data. Furthermore, the client obtains the intermediate rendering results obtained during the rendering process of the updated reconstructed scene and uses them to generate a virtual reference image. This virtual reference image is then combined with the virtual reference image to decode the reconstructed image from the code stream.

[0428] That is, the scene data of the current frame and the rendered image encoded using the virtual reference image are transmitted between the cloud platform and the client. Since the virtual reference image has a higher similarity with the rendered image of the current frame, the virtual reference image can provide a more accurate reference for encoding the rendered image of the current frame, which can improve the accuracy of inter-frame prediction, thereby improving the encoding efficiency and reducing the bit rate when encoding the rendered image; at the same time, the client only needs to complete part of the image rendering task and obtain the intermediate rendering result of the current frame, which can effectively utilize the client's image rendering capabilities. In addition, in the end-cloud collaboration solution provided in the embodiment of the present application, the client does not need to download the complete data of the three-dimensional virtual scene in advance. It only needs to obtain the scene data required for the current frame during the operation of the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience.

[0429] Taking the game scene as an example, in the above-mentioned end-cloud collaborative encoding and decoding solution, the operator's scene interaction information is the player's operation information, the scene data of the current frame is the game assets required to render the current frame game scene, the local scene data is the relevant data of the newly added three-dimensional model in the current frame game scene, and the scene change data is the status change data of the three-dimensional model in the current frame game scene.

[0430] In combination with the end-cloud collaboration solution provided in the embodiment of the present application, a new game mode, namely the end-cloud collaborative game mode, can be provided in the game scene, and the end game mode, cloud game mode and end-cloud collaborative game mode can be switched among each other.

[0431] As an example, see Figure 11, which is a schematic diagram of a game mode switching method provided by an embodiment of the present application. When the game starts, the operator of the client (i.e., the player of the game) can manually select the game mode on the client. When the player selects the game mode, the client sends information such as the game mode selected by the player, the player's operation, and the client's image rendering capabilities to the cloud platform. Among them, if the player does not select a game mode, the default game mode can be the end-cloud collaborative game mode.

[0432] The cloud platform receives information from the client, determines the game mode selected by the player, and prioritizes running the game in the player's selected game mode. However, considering that clients have different image rendering capabilities, clients with weaker image rendering capabilities cannot run the end-to-end game mode, or even the end-to-cloud collaborative game mode. Therefore, after receiving information from the client, the cloud platform can determine whether the client supports the end-to-cloud collaborative game mode and the end-to-end game mode based on the client's image rendering capabilities. If the client supports it, it will determine the game mode to run based on the player's selection.

[0433] In addition, during the game operation, the cloud platform can also detect the network bandwidth between the client and the client in real time, and switch between the terminal game mode, the terminal cloud collaborative game mode, and the cloud game mode based on the network bandwidth, the client's image rendering capabilities, player selection, etc. The embodiments of this application do not limit this.

[0434] It should be noted that Figure 11 only uses player selection, client image rendering capabilities, and network bandwidth as examples for switching basis. In actual applications, other switching basis can also be used to determine whether to switch the game mode. The embodiments of this application do not limit this.

[0435] See Figure 12, which is a schematic diagram of an encoding and decoding process for end-edge collaboration provided by an embodiment of the present application. In this example, the edge device that communicates with the client and collaborates to run a three-dimensional virtual scene can be a local server or an edge server in the cloud platform, such as the target server with the closest communication distance to the client. Figure 12 only illustrates the edge device as a local server and does not constitute a limitation on the location of the edge device.

[0436] Considering that in actual applications, when multiple clients access the same 3D virtual scene, the operations triggered by the operators of each client within the 3D virtual scene may affect each other, running the 3D virtual scene requires extremely low latency and high network requirements. In this case, the data of the 3D virtual scene can be downloaded to a local server, which then executes the encoding method shown in Figure 7. Based on the scene data of the current frame, the local server renders and generates a rendered image and an intermediate rendering result of the current frame. The intermediate rendering result is then used to generate a virtual reference image, which is then encoded using the virtual reference image. The code stream containing the rendered image and scene data of the current frame is then sent to multiple clients.

[0437] After receiving the code stream sent by the local server, the client executes the decoding method shown in Figure 9. Based on the scene data of the current frame obtained by decoding, it renders and generates the rendering intermediate result of the current frame, then uses the rendering intermediate result to generate a virtual reference image, and uses the virtual reference image for auxiliary decoding to obtain the rendered image reconstructed for the current frame.

[0438] In a possible implementation, the local server may also encode the rendered image of the current frame obtained by rendering and transmit the encoded rendered image of the current frame to multiple clients for decoding and display in a video streaming manner.

[0439] In the above-mentioned end-edge collaboration solution, each client and local server in the same LAN environment is connected to the same network, the impact of network fluctuations is minimal, and the physical distance between the local server and the client is smaller, making the delay in transmitting the code stream between the local server and the client lower than that of the end-cloud collaboration solution.

[0440] See Figure 13, which is a schematic diagram of a coding and decoding process for end-edge-cloud collaboration provided by an embodiment of the present application. In this example, the edge device communicating with the client can be an edge server in the cloud platform or a local server of the client; and the client has certain image rendering capabilities.

[0441] Among them, the central server of the cloud platform executes the encoding method shown in Figure 7, and based on the scene data of the current frame, renders and generates the rendered image and rendering intermediate results of the current frame, then uses the rendering intermediate results to generate a virtual reference image, and uses the virtual reference image to encode the rendered image, and then sends the code stream encoded with the rendered image and scene data of the current frame to the edge device.

[0442] After receiving the code stream from the central server, the edge device executes the decoding method shown in Figure 9. Based on the decoded scene data of the current frame, it renders and generates an intermediate rendering result for the current frame. It then uses this intermediate rendering result to generate a virtual reference image, which is then used for auxiliary decoding to obtain a reconstructed rendered image for the current frame. Furthermore, the edge device can generate a depth map for the current frame during the rendering process. Finally, the edge device encodes and sends the reconstructed rendered image and depth map to the client.

[0443] Among them, when encoding the reconstructed image of the current frame, the edge device can directly encode the reconstructed rendering image of the current frame into the code stream, or it can encode the residual between the current frame and the reconstructed rendering image of the historical frame into the code stream based on the reconstructed image of the historical frame. The embodiment of the present application does not impose any restrictions on this.

[0444] After receiving the code stream sent by the edge device, the client decodes the depth map and the rendered image reconstructed from the current frame, and generates the final display image based on the depth map and the reconstructed rendered image.

[0445] In the above-mentioned end-edge-cloud collaborative solution, for clients with weak image rendering capabilities such as VR, the client's data processing volume can be further reduced. The client only needs to complete a small amount of synthesis calculation of depth information and reconstructed rendered images, as well as display.

[0446] It should be noted that the above Figures 10 to 13 are only for the purpose of clearly illustrating the encoding and decoding method provided in the embodiments of the present application, and the several combination schemes / exemplary implementation schemes provided. Of course, based on the inventive concept of the scheme of the present application, a variety of similar combination schemes / implementation schemes can be extended. The embodiments of the present application are not listed here one by one. When they are the same as the inventive concept of the scheme of the present application, each implementation scheme or combination scheme in the above embodiments should be included in the scope of protection of this application.

[0447] FIG14 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application. The encoding device can be implemented as part or all of the above-mentioned encoder by software, hardware, or a combination of both. Referring to FIG14 , the encoding device includes: a data acquisition module 1401, a data encoding module 1402, and an image encoding module 1403.

[0448] The data acquisition module 1401 is used to acquire the scene data of the current frame in the three-dimensional virtual scene; the detailed implementation process refers to the corresponding content in the above step 701 and will not be repeated here.

[0449] The data encoding module 1402 is used to encode the scene data into the bitstream; the detailed implementation process refers to the corresponding content in the above step 702 and will not be repeated here.

[0450] The image encoding module 1403 is used to encode the rendered image of the current frame into a bitstream based on the scene data. The detailed implementation process refers to the corresponding content in the above step 703 and will not be repeated here.

[0451] In a possible implementation, the image encoding module 1403 includes:

[0452] A rendering result acquisition unit is used to obtain the rendering intermediate result of the current frame, where the rendering intermediate result is obtained by rendering the scene data;

[0453] The image encoding unit is configured to encode the rendered image based on the intermediate rendering result.

[0454] In a possible implementation, the image encoding unit is specifically configured to:

[0455] Generate a virtual reference image based on the intermediate rendering results;

[0456] Based on the virtual reference image, the rendered image is encoded into the bitstream.

[0457] In a possible implementation, the rendering intermediate result includes a depth map, and the image encoding unit is specifically configured to:

[0458] Based on the depth map, generating partition information of a first image block, where the first image block is an image block of a rendered image, and the partition information is used to divide the first image block into a plurality of partitions;

[0459] Encode the partition information into the code stream.

[0460] In a possible implementation, the image encoding unit is further configured to:

[0461] Based on the depth map, determine a second segmentation information set from the first segmentation information set, where the first segmentation information set includes a plurality of segmentation information, and the second segmentation information set is a subset of the first segmentation information set;

[0462] Partition information of the first image block is determined from the second partition set.

[0463] In a possible implementation, the image encoding unit is further configured to:

[0464] Based on the depth map, determining a location of a depth change in the first image block;

[0465] Based on the position, a second partition information set is determined from the first partition information set.

[0466] In a possible implementation, the rendering intermediate result includes a graphics motion vector of the current frame; and the image encoding unit includes:

[0467] The MV encoding subunit is used to determine the motion vector MV of the second image block based on the graphic motion vector, and encode the MV of the second image block; the second image block is any image block in the rendered image.

[0468] In a possible implementation, the MV encoding subunit is specifically configured to:

[0469] Determine a predicted motion vector MV of a key pixel point in the second image block based on the graphic motion vector, where the key pixel point includes one or more pixel points;

[0470] Determine the search range based on the predicted MV of the key pixel points;

[0471] Determining, according to the search range, a matching block corresponding to the second image block from a reference rendered image, where the reference rendered image is a rendered image of a reference frame corresponding to the current frame;

[0472] An MV of the second image block is determined based on the second image block and a matching block corresponding to the second image block.

[0473] In a possible implementation, the MV encoding subunit is specifically configured to:

[0474] Determine a predicted motion vector MV of a key pixel point in the second image block based on the graphic motion vector, where the key pixel point includes one or more pixel points;

[0475] Determining a predicted MV of the second image block based on the predicted MV of the key pixel point;

[0476] Based on the predicted MV, the MV of the second image block is encoded.

[0477] In a possible implementation, the MV encoding subunit is further configured to:

[0478] Constructing a candidate MV list of the second image block, where the candidate MV list includes predicted MVs of key pixels;

[0479] Determining a predicted MV of the second image block based on the candidate MV list;

[0480] In a possible implementation, the MV encoding subunit is further configured to:

[0481] Index information is encoded into the bitstream, where the index information indicates a position of the predicted MV of the second image block in the candidate MV list.

[0482] In a possible implementation, the MV encoding subunit is further configured to:

[0483] Obtaining an MV residual based on the MV of the second image block and the predicted MV of the second image block;

[0484] Encode the MV residual into the bitstream.

[0485] In one possible implementation, the scene data includes local scene data and / or scene change data. The local scene data refers to the scene data newly added to the current frame relative to the historical frame, and the scene change data refers to the scene data whose state has changed relative to the historical frame.

[0486] In one possible implementation, the encoding device is included in the target server, and the data acquisition module 1401 includes:

[0487] An interactive information receiving unit, configured to receive scene interactive information sent by a client, the scene interactive information indicating an operation triggered by an operator in the three-dimensional virtual scene and the position of the operator in the three-dimensional virtual scene;

[0488] The data determination unit is used to determine the scene data of the current frame based on the scene interaction information.

[0489] In a possible implementation, the data determination unit is specifically configured to:

[0490] Based on the scene interaction information, the scene data within the target range is determined as the scene data of the current frame. The target range refers to the range of the target size extending from the operator's position in the three-dimensional virtual scene as the center point; or

[0491] Based on the scene interaction information, the position and posture of the virtual camera in the three-dimensional virtual scene are determined. Based on the position and posture of the virtual camera, the scene data within the visible range of the virtual camera is determined as the scene data of the current frame.

[0492] In a possible implementation, the encoding device is included in the target server, and the encoding device further includes:

[0493] The information acquisition module is used to obtain the client's device information, which indicates the client's image rendering capabilities. If it is determined based on the client's device information that the three-dimensional virtual scene requires end-cloud collaboration, the step of obtaining scene data of the current frame in the three-dimensional virtual scene is executed.

[0494] In a possible implementation, the encoding device is included in the target server, and the encoding device further includes:

[0495] The network detection module is used to obtain the network bandwidth between the target server and the client; if the three-dimensional virtual scene is not currently running in end-cloud collaboration and the network bandwidth meets the switching conditions for end-cloud collaboration, the step of obtaining scene data of the current frame in the three-dimensional virtual scene is executed.

[0496] In a possible implementation, the target server is a server in the cloud platform with the shortest communication distance to the client.

[0497] In an embodiment of the present application, scene data is transmitted via a code stream, so that the decoding end (or the end receiving the code stream) does not need to pre-download and store the scene data required to run the three-dimensional virtual scene. At the same time, when transmitting the scene data of the current frame, the scene data of the encoding end and the decoding end can remain consistent. Based on this, the encoding end can effectively utilize the scene data in the process of encoding the rendered image of the current frame, and optimize the encoding process of the rendered image to reduce the bit rate and computational overhead of the encoded rendered image. Furthermore, when using scene data to encode the rendered image can reduce the bit rate, this encoding method can also reduce the bandwidth resources for transmitting the code stream between the encoding end and the decoding end.

[0498] In addition, for the game scene, the encoding end of the encoding method provided by the present application can be a cloud platform, and the corresponding decoding end can be a client, that is, the cloud platform and the client use an end-cloud collaborative approach to run the three-dimensional virtual scene. After the cloud platform determines the scene data of the current frame, it can encode the scene data into a code stream and send it to the client. In this way, the client does not need to pre-download and store the scene data required to run the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience. In the case where the cloud platform and the client have the same scene data, the cloud platform can effectively use the scene data in the process of encoding the rendered image of the current frame and optimize the encoding process of the rendered image; correspondingly, the client can decode the code stream based on the same scene data and process the decoding information in a corresponding manner to obtain the rendered image reconstructed by the current frame. In this way, not only can the bit rate and computational overhead of the cloud platform's encoded rendered image be reduced, but the client's own processing power can also be effectively utilized, thereby reducing the bandwidth resources for transmitting the code stream between the cloud platform and the client.

[0499] It should be noted that the encoding device provided in the above embodiment is merely illustrated by the division of the above functional modules during encoding. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the encoding device provided in the above embodiment and the encoding method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0500] FIG15 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application. The decoding device can be implemented by software, hardware, or a combination of both to form part or all of the above-mentioned decoder. Referring to FIG15 , the decoding device includes: a data decoding module 1501 and an image reconstruction module 1502.

[0501] The data decoding module 1501 is used to decode the code stream to obtain the scene data of the current frame in the three-dimensional virtual scene; the detailed implementation process refers to the corresponding content in the above step 901 and will not be repeated here.

[0502] The image reconstruction module 1502 is used to decode the code stream based on the scene data to obtain a rendered image reconstructed for the current frame. The detailed implementation process is referred to the corresponding content in the above step 902 and will not be repeated here.

[0503] In a possible implementation, the image reconstruction module 1502 further includes:

[0504] A rendering result acquisition unit is used to obtain the rendering intermediate result of the current frame, where the rendering intermediate result is obtained by rendering the scene data;

[0505] The image reconstruction unit is used to decode the code stream based on the intermediate rendering result to obtain a rendered image reconstructed for the current frame.

[0506] In a possible implementation, the image reconstruction unit is specifically configured to:

[0507] Generate a virtual reference image based on the intermediate rendering results;

[0508] The code stream is decoded based on the virtual reference image to obtain a rendered image reconstructed for the current frame.

[0509] In a possible implementation, the intermediate rendering result includes a graphics motion vector of the current frame, and the image reconstruction unit is specifically configured to:

[0510] The MV determination subunit is configured to decode the code stream based on the graphic motion vector to obtain a motion vector MV of a second image block, where the second image block is any image block in the rendered image.

[0511] In a possible implementation, the MV determination subunit is further configured to:

[0512] Determine a predicted MV of a key pixel point in the second image block based on the graphic motion vector, where the key pixel point includes one or more pixel points;

[0513] Constructing a candidate MV list of the second image block, where the candidate MV list includes predicted MVs of key pixels;

[0514] The MV of the second image block is obtained from the candidate MV list based on index information, where the index information is obtained from the code stream and indicates a position of the MV of the second image block in the candidate motion vector list.

[0515] In a possible implementation, the MV determination subunit is further configured to:

[0516] Determine a predicted MV of a key pixel point in the second image block based on the graphic motion vector, where the key pixel point includes one or more pixel points;

[0517] Constructing a candidate MV list of the second image block, where the candidate MV list includes predicted MVs of key pixels;

[0518] Obtaining a candidate MV of the second image block from the candidate MV list based on index information, where the index information is obtained from the bitstream and the index information indicates a position of the MV of the second image block in the candidate MV list;

[0519] The MV of the second image block is obtained based on the candidate MV of the second image block and the MV residual corresponding to the second image block, where the MV residual is obtained from the bitstream.

[0520] In one possible implementation, the scene data includes local scene data and / or scene change data. The local scene data refers to the scene data newly added to the current frame relative to the historical frame, and the scene change data refers to the scene data whose state has changed relative to the historical frame.

[0521] In a possible implementation, the scene data includes local scene data and scene change data; and the rendering result acquisition unit includes:

[0522] A scene reconstruction subunit is used to reconstruct the scene based on the local scene data and the scene data of the historical frames to obtain the reconstructed scene of the current frame;

[0523] A scene update subunit, configured to update the reconstructed scene of the current frame based on the scene change data;

[0524] The data acquisition subunit is used to obtain the intermediate rendering result of the current frame based on the updated reconstructed scene of the current frame.

[0525] In a possible implementation, the decoding device is included in the client, and the decoding device further includes:

[0526] The information sending module is used to send scene interaction information to the target server. The scene interaction information indicates the operation triggered by the operator in the three-dimensional virtual scene and the position of the operator in the three-dimensional virtual scene.

[0527] In a possible implementation, the decoding device further includes:

[0528] The mode acquisition module is used to obtain the operating mode of the three-dimensional virtual scene; if the operating mode is end-cloud collaborative operation, the step of sending scene interaction information to the target server in the cloud platform is executed.

[0529] In a possible implementation, the target server is a server in the cloud platform with the shortest communication distance to the client.

[0530] In an embodiment of the present application, the cloud platform and the client use a terminal-cloud collaborative mode to run a three-dimensional virtual scene. The cloud platform sends the scene data of the current frame and the rendered image of the current frame encoded using the scene data to the client via a code stream. The client, in combination with its own image rendering capabilities, uses the scene data of the current frame for auxiliary decoding to obtain a rendered image reconstructed from the current frame. In this process, the client does not need to pre-download and store the scene data required to run the three-dimensional virtual scene, which reduces the requirements for the client to run the three-dimensional virtual scene and improves the user experience. Moreover, if the scene data of the cloud platform and the client can be consistent, the cloud platform can effectively utilize the scene data during the encoding process of the rendered image of the current frame and optimize the encoding process of the rendered image to reduce the bit rate and computational overhead of the encoded rendered image. Furthermore, if using scene data to encode the rendered image can reduce the bit rate, this encoding method can also reduce the bandwidth resources required to transmit the code stream between the encoding and decoding ends. Correspondingly, after parsing the scene data of the current frame, the client can decode the code stream based on the same scene data and process the decoding information in a corresponding manner to obtain a rendered image reconstructed from the current frame. This not only reduces the bit rate and computational overhead of encoding and rendering images on the cloud platform, but also effectively utilizes the client's own processing power, thereby reducing the bandwidth resources for transmitting code streams between the cloud platform and the client.

[0531] It should be noted that the decoding device provided in the above embodiment is merely illustrated by the division of the above functional modules during decoding. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the decoding device provided in the above embodiment and the decoding method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0532] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the steps of the encoding method shown in the above embodiment, or execute the steps of the decoding method shown in the above embodiment.

[0533] An embodiment of the present application also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or processor executes the steps of the encoding method shown in the above embodiment, or executes the steps of the decoding method shown in the above embodiment.

[0534] An embodiment of the present application also provides a computer program, which, when executed on a computer or processor, enables the computer or processor to execute the steps of the encoding method shown in the above embodiment, or to execute the steps of the decoding method shown in the above embodiment.

[0535] An embodiment of the present application also provides a coding and decoding system, which includes a coding device and a decoding device. The coding device is used to implement the steps of the coding method shown in the above embodiment, and the decoding device is used to implement the steps of the decoding method shown in the above embodiment.

[0536] An embodiment of the present application further provides an encoder, which includes a processing circuit, and the processing circuit is used to implement the encoding method shown in the above embodiment.

[0537] An embodiment of the present application further provides a decoder, which includes a processing circuit, and the processing circuit is used to implement the decoding method shown in the above embodiment.

[0538] The embodiment of the present application further provides an encoded code stream, which is generated according to the encoding method shown in the above embodiment.

[0539] An embodiment of the present application further provides a computer-readable storage medium, which stores a code stream generated according to the encoding method shown in the above embodiment.

[0540] An embodiment of the present application further provides a device for storing a code stream, the device comprising a receiver and at least one storage medium, the receiver being configured to receive a code stream generated according to the encoding method shown in the above embodiment, and the at least one storage medium being configured to store the code stream.

[0541] An embodiment of the present application also provides a device for transmitting a code stream, which includes a transmitter and a receiver. The receiver is used to receive the code stream generated according to the encoding method shown in the above embodiment, and the transmitter is used to send the code stream to the terminal side device through a transmission medium.

[0542] An embodiment of the present application also provides a device for transmitting a code stream, which includes a transmitter and at least one storage medium. The at least one storage medium is used to store the code stream generated according to the encoding method shown in the above embodiment. The transmitter is used to obtain the code stream from the storage medium and send the code stream to the end-side device via the transmission medium.

[0543] An embodiment of the present application also provides a system for distributing code streams, the system including at least one storage medium, the at least one storage medium being used to store code streams generated according to the encoding method shown in the above embodiment, the streaming media device being used to obtain a target code stream from the at least one storage medium and send the target code stream to a terminal-side device, wherein the streaming media device includes a content server or a content distribution server.

[0544] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0545] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0546] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the scene data of the current frame in the three-dimensional virtual scene, the client's device information, and the scene interaction information involved in the embodiments of this application are all obtained with full authorization.

[0547] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A coding method, characterized in that: The method comprises: Obtain scene data of the current frame in the three-dimensional virtual scene; Encoding the scene data into a bitstream; Based on the scene data, the rendered image of the current frame is encoded into the code stream.

2. The method according to claim 1, wherein The step of encoding the rendered image of the current frame into the bitstream based on the scene data includes: Obtaining an intermediate rendering result of the current frame, where the intermediate rendering result is obtained by rendering the scene data; The rendered image is encoded based on the intermediate rendering result.

3. The method according to claim 2, wherein The encoding of the rendered image based on the intermediate rendering result includes: generating a virtual reference image based on the intermediate rendering result; The rendered image is encoded into a bitstream based on the virtual reference image.

4. The method according to claim 2, wherein The rendering intermediate result includes a depth map, and encoding the rendered image based on the rendering intermediate result includes: generating, based on the depth map, partition information of a first image block, where the first image block is an image block of the rendered image, the partition information being used to partition the first image block into a plurality of partitions; The division information is encoded into a code stream.

5. The method according to claim 4, wherein The generating, based on the depth map, division information of the first image block includes: Determining, based on the depth map, a second partition information set from a first partition information set, where the first partition information set includes a plurality of partition information, and the second partition information set is a subset of the first partition information set; The partition information of the first image block is determined from the second partition set.

6. The method according to claim 5, wherein The determining, based on the depth map, a second segmentation information set from the first segmentation information set includes: determining, based on the depth map, a location of a depth change in the first image block; Based on the position, the second partition information set is determined from the first partition information set.

7. The method according to claim 2, wherein The rendering intermediate result includes the graphics motion vector of the current frame; The encoding of the rendered image based on the intermediate rendering result includes: Based on the graphic motion vector, a motion vector MV of a second image block is determined, and the MV of the second image block is encoded; the second image block is any image block in the rendered image.

8. The method according to claim 7, wherein The determining the motion vector MV of the second image block based on the graphic motion vector includes: Determine, based on the graphic motion vector, a predicted MV of a key pixel in the second image block, where the key pixel includes one or more pixels; Determining a search range based on the predicted MV of the key pixel point; Determining, according to the search range, a matching block corresponding to the second image block from a reference rendered image, where the reference rendered image is a rendered image of a reference frame corresponding to the current frame; The MV of the second image block is determined based on the second image block and a matching block corresponding to the second image block.

9. The method according to claim 7 or 8, wherein The encoding of the MV of the second image block includes: Determine, based on the graphic motion vector, a predicted MV of a key pixel in the second image block, where the key pixel includes one or more pixels; Determining a predicted MV of the second image block based on the predicted MV of the key pixel point; Based on the predicted MV, the MV of the second image block is encoded.

10. The method according to claim 9, wherein Determining the predicted motion vector MV of the second image block based on the predicted MV of the key pixel point includes: Constructing a candidate MV list of the second image block, wherein the candidate MV list includes the predicted MV of the key pixel point; Based on the candidate MV list, a predicted MV of the second image block is determined.

11. The method according to any one of claims 1 to 10, wherein: The scene data includes local scene data and / or scene change data. The local scene data refers to scene data newly added to the current frame relative to the historical frame, and the scene change data refers to scene data whose state has changed relative to the historical frame.

12. The method according to any one of claims 1 to 11, wherein: The method is applied to a target server, and acquiring scene data of a current frame in a three-dimensional virtual scene includes: receiving scene interaction information sent by a client, wherein the scene interaction information indicates an operation triggered by an operator in the three-dimensional virtual scene and a position of the operator in the three-dimensional virtual scene; Based on the scene interaction information, scene data of the current frame is determined.

13. The method according to claim 12, wherein: The determining the scene data of the current frame based on the scene interaction information includes: Based on the scene interaction information, scene data within a target range is determined as the scene data of the current frame, where the target range refers to a range with a target size extending from the position of the operator in the three-dimensional virtual scene as the center point; or Based on the scene interaction information, the position and posture of the virtual camera in the three-dimensional virtual scene are determined; based on the position and posture of the virtual camera, the scene data within the visible range of the virtual camera is determined as the scene data of the current frame.

14. The method according to any one of claims 1 to 13, wherein: The method is applied to a target server, and before acquiring scene data of a current frame in a three-dimensional virtual scene, the method further includes: Acquire device information of the client, wherein the device information includes image rendering capability of the client; The end-cloud collaboration mode is determined based on the device information of the client.

15. The method according to any one of claims 1 to 13, wherein: The method is applied to a target server, and before acquiring scene data of a current frame in a three-dimensional virtual scene, the method further includes: Obtaining the network bandwidth between the target server and the client; The end-cloud collaboration mode is determined based on the network bandwidth.

16. A decoding method, characterized in that: The method comprises: Obtain scene data of the current frame in the three-dimensional virtual scene from the code stream; The code stream is decoded based on the scene data to obtain a rendered image reconstructed from the current frame.

17. The method according to claim 16, wherein The decoding the code stream based on the scene data includes: Obtaining an intermediate rendering result of the current frame, where the intermediate rendering result is obtained by rendering the scene data; The code stream is decoded based on the intermediate rendering result to obtain a rendered image reconstructed from the current frame.

18. The method according to claim 17, wherein The decoding the code stream based on the rendering intermediate result includes: generating a virtual reference image based on the intermediate rendering result; The code stream is decoded based on the virtual reference image to obtain a rendered image reconstructed from the current frame.

19. The method according to claim 17, wherein The rendering intermediate result includes a graphics motion vector of the current frame, and decoding the code stream based on the rendering intermediate result includes: The code stream is decoded based on the graphic motion vector to obtain a motion vector MV of a second image block, where the second image block is any image block in the rendered image.

20. The method according to claim 19, wherein The decoding the code stream based on the graphic motion vector includes: Determine, based on the graphic motion vector, a predicted MV of a key pixel in the second image block, where the key pixel includes one or more pixels; Constructing a candidate MV list of the second image block, wherein the candidate MV list includes the predicted MV of the key pixel point; The MV of the second image block is obtained from the candidate MV list based on index information, where the index information is obtained from the code stream, and the index information indicates a position of the MV of the second image block in the candidate motion vector list.

21. The method according to claim 19, wherein The decoding the code stream based on the graphic motion vector includes: Determine, based on the graphic motion vector, a predicted MV of a key pixel in the second image block, where the key pixel includes one or more pixels; Constructing a candidate MV list of the second image block, wherein the candidate MV list includes the predicted MV of the key pixel point; Obtaining a candidate MV of the second image block from the candidate MV list based on index information, where the index information is obtained from the bitstream and the index information indicates a position of the MV of the second image block in the candidate MV list; The MV of the second image block is obtained based on the candidate MV of the second image block and the MV residual corresponding to the second image block, where the MV residual is obtained from the code stream.

22. The method according to any one of claims 17 to 21, wherein: The scene data includes local scene data and / or scene change data. The local scene data refers to scene data newly added to the current frame relative to the historical frame, and the scene change data refers to scene data whose state has changed relative to the historical frame.

23. The method according to claim 22, wherein The scene data includes the local scene data and the scene change data; and obtaining the intermediate rendering result of the current frame includes: Performing scene reconstruction based on the local scene data and the scene data of the historical frame to obtain a reconstructed scene of the current frame; updating the reconstructed scene of the current frame based on the scene change data; Based on the updated reconstructed scene of the current frame, an intermediate rendering result of the current frame is obtained.

24. The method according to any one of claims 16 to 23, wherein: The method is applied to a client; before decoding the code stream to obtain scene data of the current frame in the three-dimensional virtual scene, the method further includes: Scene interaction information is sent to a target server, where the scene interaction information indicates an operation triggered by an operator in the three-dimensional virtual scene and a position of the operator in the three-dimensional virtual scene.

25. The method of claim 24, wherein: Before sending the scene interaction information to the target server, the method further includes: Acquiring an operating mode of the three-dimensional virtual scene; If the operation mode is end-cloud collaborative operation, the step of sending scene interaction information to the target server is executed.

26. An encoding device, characterized in that The device is contained in a target server in a cloud platform and includes: A data acquisition module is used to acquire scene data of the current frame in the three-dimensional virtual scene; An image encoding module, configured to encode the rendered image of the current frame into a bitstream based on the scene data; The data encoding module is used to encode the scene data into a code stream.

27. A decoding device, characterized in that: The device comprises: A data decoding module is used to decode the code stream to obtain scene data of the current frame in the three-dimensional virtual scene; An image reconstruction module is configured to determine a rendered image reconstructed from the current frame based on the scene data and the code stream; wherein the rendered image is encoded into the code stream based on the scene data.

28. An encoding device, characterized in that The encoding device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 15.

29. A decoding device, characterized in that The decoding device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 16 to 25.

30. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when run on a computer or a processor, enables the computer or the processor to execute the steps of the method according to any one of claims 1 to 15, or to execute the steps of the method according to any one of claims 16 to 25.

31. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 15 are performed, or the steps of the method according to any one of claims 16 to 25 are performed.

32. A coded bitstream, characterized in that The code stream is generated according to the method according to any one of claims 1 to 15.

33. A coded bitstream, characterized in that The code stream includes scene data of a current frame in a three-dimensional virtual scene and a rendered image of the current frame. The rendered image is encoded into the code stream based on the scene data.

34. A computer-readable storage medium, characterized in that The computer-readable storage medium stores the code stream according to any one of claims 1-15.

35. A device for storing a code stream, characterized in that: The device comprises: a receiver and at least one storage medium, The receiver is used to receive the code stream according to any one of claims 1 to 15; The at least one storage medium is used to store the code stream.

36. A device for transmitting a code stream, characterized in that: The device comprises: a transmitter and a receiver, The receiver is used to receive the code stream according to any one of claims 1 to 15; The transmitter is used to transmit the code stream.

37. A device for transmitting a code stream, characterized in that: The device includes: a transmitter and at least one storage medium, The at least one storage medium is used to store the code stream according to any one of claims 1 to 15; The transmitter is used to obtain the code stream from the storage medium and transmit the code stream.

38. A system for distributing code streams, characterized in that: The system comprises: At least one storage medium, configured to store at least one code stream according to any one of claims 1 to 15; The streaming media device is configured to obtain a target code stream from the at least one storage medium and send the target code stream to the terminal side device, wherein the streaming media device includes a content server or a content distribution server.

Images