Three-dimensional video processing method, apparatus, device, and storage medium

Abstract

This application provides a three-dimensional video processing method, apparatus, device, and storage medium, relating to the field of computer vision technology. The method involves quantizing and encoding a keyframe representation model to obtain a keyframe encoded model, then obtaining a keyframe decoding model. Non-keyframe sequences are selected one by one as processing frame sequences. A reference frame sequence is obtained based on the previous acquisition time of the processing frame sequence. Transform parameters are generated based on the decoding model corresponding to the reference frame sequence. These transform parameters are quantized and encoded to obtain encoded transform parameters, which are then sent to the decoding end. If a residual identifier exists, a residual representation model is generated based on the transform parameters. This residual representation model is quantized and encoded, and then sent to the decoding end. These transform parameters are used to indicate the data change trend between previous and subsequent acquisition times. The residual representation model is used to maximize the offsetting of temporal errors, thereby preventing errors from accumulating over time.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to three-dimensional video processing methods, apparatus, devices and storage media. Background Technology

[0002] 3D Gaussian Splatting (3DGS) surpasses earlier 3D scene representation techniques such as point clouds, meshes, and Neural Radiation Fields (NeRF) in terms of 3D scene reconstruction quality, reconstruction speed, interactive freedom, and rendering speed, and has become the widely used 3D scene representation method today.

[0003] Among related technologies, the 3D Gaussian splashing technique utilizes neural networks to model the rotation and translation changes of Gaussian points at adjacent time points, using inter-frame prediction results as a reference for the next frame for 3D reconstruction. However, this approach suffers from temporal error accumulation, affecting the reconstruction performance of 3D video. Summary of the Invention

[0004] The main objective of this application is to propose a three-dimensional video processing method, apparatus, device, and storage medium to reduce temporal error accumulation and improve the reconstruction performance of three-dimensional video.

[0005] To achieve the above objectives, a first aspect of this application proposes a three-dimensional video processing method applied at the encoding end, the method comprising:

[0006] At multiple acquisition times, video frames corresponding to the target scene from different perspectives are acquired. The video frames corresponding to the initial acquisition time are used as the key frame sequence, and the video frames corresponding to other acquisition times are used as non-key frame sequences.

[0007] A keyframe representation model is generated based on the keyframe sequence. The keyframe representation model is then quantized and encoded to obtain a keyframe encoding model. Finally, the keyframe encoding model is decoded to obtain a keyframe decoding model.

[0008] One by one, the non-keyframe sequences are selected as processing frame sequences. A reference frame sequence is obtained based on the previous acquisition time of the processing frame sequence. Transformation parameters are generated based on the decoding model corresponding to the reference frame sequence. The transformation parameters are quantized and encoded to obtain encoded transformation parameters. The encoded transformation parameters are sent to the decoding end. If a residual identifier exists, a residual representation model corresponding to the processing frame sequence is generated based on the transformation parameters. The residual representation model is quantized and encoded to obtain a residual encoding model. The residual encoding model is sent to the decoding end. The decoding model corresponding to the first processing frame sequence is the keyframe decoding model.

[0009] In one embodiment, the step of performing quantization encoding on the keyframe representation model to obtain a keyframe encoding model includes:

[0010] Obtain the quantization step size, and perform quantization operation on the first model parameters of the keyframe representation model according to the quantization step size to obtain the quantized representation model;

[0011] Entropy coding is performed on the quantization representation model to obtain the keyframe coding model.

[0012] In one embodiment, obtaining the quantization step size includes:

[0013] Obtain the first center position of multiple three-dimensional Gaussian distributions in the keyframe representation model;

[0014] The first center position is input into a hash list to obtain the first context feature, and the first context feature is input into a quantization prediction model for data processing to obtain the quantization step size.

[0015] In one embodiment, generating transform parameters based on the decoding model includes:

[0016] Obtain the second center positions of multiple three-dimensional Gaussian distributions in the decoding model;

[0017] The second center position is input into a hash list to obtain the second context feature, and the second context feature is input into a motion transformation prediction model for data processing to obtain the transformation parameters. The transformation parameters include the change in the prediction parameters of the second model parameters corresponding to the three-dimensional Gaussian distribution in the decoding model at the acquisition time corresponding to the processing frame sequence.

[0018] In one embodiment, generating the residual representation model corresponding to the processed frame sequence based on the transform parameters includes:

[0019] Based on the transformation parameters, the decoding model is subjected to inter-frame transformation to obtain a reference decoding model;

[0020] The residual identifier is generated based on the reference decoding model;

[0021] The residual representation model is generated based on the residual identifier.

[0022] In one embodiment, performing inter-frame transformation on the decoding model according to the transformation parameters to obtain a reference decoding model includes:

[0023] At least the center position parameter, covariance parameter, color parameter, and opacity parameter corresponding to each three-dimensional Gaussian distribution of the decoding model are obtained;

[0024] Based on the transformation parameters, the center position update parameters corresponding to the center position parameter, the covariance update parameters corresponding to the covariance parameter, the color update parameters corresponding to the color parameter, and the opacity update parameters corresponding to the opacity parameter are obtained at the acquisition time corresponding to the processing frame sequence.

[0025] The reference decoding model is obtained by updating the three-dimensional Gaussian distribution based on the center position update parameter, the covariance update parameter, the color update parameter, and the opacity update parameter.

[0026] In one embodiment, generating the residual identifier based on the reference decoding model includes:

[0027] The reference decoding model is rendered to obtain the reference rendered image corresponding to the processed frame sequence;

[0028] The residual identifier is generated based on the distortion parameters between the reference rendered image and the video frames corresponding to the processed frame sequence.

[0029] In one embodiment, generating the residual representation model includes:

[0030] The residual representation region of the processed frame sequence is determined based on the residual identifier;

[0031] Generate at least one three-dimensional Gaussian distribution for the residual representation region to form the residual representation model.

[0032] In one embodiment, the method further includes:

[0033] Obtain the rendering distortion loss term corresponding to the keyframe decoding model or the non-keyframe decoding model;

[0034] Obtain the bitrate loss term, which is calculated by the decoding model or the residual representation model during the quantization encoding process;

[0035] A rate-distortion joint loss term is calculated based on the rendering distortion loss term and the bitrate loss term, and the rate-distortion joint loss term is used to optimize at least the decoding model, the residual representation model, and the transform parameters.

[0036] To achieve the above objectives, a second aspect of this application proposes a three-dimensional video processing method applied at a decoding end, the method comprising:

[0037] The acquired keyframe encoding model is decoded to obtain the keyframe decoding model;

[0038] For each processed frame sequence, the corresponding encoding transformation parameters are obtained, and the decoding transformation parameters are obtained based on the encoding transformation parameters. Inter-frame transformation is performed using the decoding transformation parameters and the decoding model corresponding to the reference frame sequence to obtain a reference decoding model. The reference decoding model is used as the non-keyframe decoding model. If the processed frame sequence includes a residual encoding model, the residual encoding model is decoded to obtain a residual decoding model. The residual decoding model and the reference decoding model are residually fused to obtain the non-keyframe decoding model. The decoding model of the first processed frame sequence is the keyframe decoding model.

[0039] In one embodiment, decoding the acquired keyframe encoding model to obtain a keyframe decoding model includes:

[0040] Entropy decoding is performed on the keyframe encoding model to obtain a quantization decoding model;

[0041] Obtain the quantization step size, and perform dequantization operation on the parameters of the quantization decoding model according to the quantization step size to obtain the keyframe decoding model.

[0042] In one embodiment, the method further includes:

[0043] The multiple three-dimensional Gaussian distributions contained in the keyframe decoding model and the non-keyframe decoding model are projected in two dimensions at each viewpoint to obtain the two-dimensional Gaussian distribution of the corresponding viewpoint.

[0044] Each of the two-dimensional Gaussian distributions is rendered from the corresponding viewpoint to obtain the target rendered image corresponding to all viewpoints at each acquisition time of the target scene;

[0045] A 3D scene corresponding to the target scene is generated based on the target rendered image.

[0046] To achieve the above objectives, a third aspect of this application provides a three-dimensional video processing apparatus applied at an encoding end, the apparatus comprising:

[0047] Frame segmentation module: used to acquire video frames of the target scene from different perspectives at multiple acquisition times, taking the video frames corresponding to the initial acquisition time as the key frame sequence and the video frames corresponding to other acquisition times as the non-key frame sequence.

[0048] Keyframe encoding module: used to generate a keyframe representation model based on the keyframe sequence, perform quantization encoding on the keyframe representation model to obtain a keyframe encoding model, and decode the keyframe encoding model to obtain a keyframe decoding model;

[0049] Non-keyframe encoding module: used to select the non-keyframe sequence one by one as the processing frame sequence, obtain the reference frame sequence according to the previous acquisition time of the processing frame sequence, generate transformation parameters according to the decoding model corresponding to the reference frame sequence, perform quantization encoding operation on the transformation parameters to obtain encoded transformation parameters, send the encoded transformation parameters to the decoding end, if there is a residual identifier, generate the residual representation model corresponding to the processing frame sequence based on the transformation parameters, perform quantization encoding operation on the residual representation model to obtain the residual encoding model, and send the residual encoding model to the decoding end. The decoding model corresponding to the first processing frame sequence is the keyframe decoding model.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0051] The three-dimensional video processing method, apparatus, device, and storage medium proposed in this application acquire video frames of the target scene from different perspectives at multiple acquisition times. The video frames at the initial acquisition time are used as a keyframe sequence, and the video frames at other acquisition times are used as non-keyframe sequences. A keyframe representation model is then generated based on the keyframe sequence. This keyframe representation model is quantized and encoded to obtain a keyframe encoding model. The keyframe encoding model is then decoded to obtain a keyframe decoding model. Next, non-keyframe sequences are selected one by one as processing frame sequences. A reference frame sequence is obtained based on the previous acquisition time of the processing frame sequence. Transformation parameters are generated based on the decoding model corresponding to the reference frame sequence. These transformation parameters are quantized and encoded to obtain encoded transformation parameters, which are then sent to the decoding end. If a residual identifier exists, a residual representation model corresponding to the processing frame sequence is generated based on the transformation parameters. This residual representation model is quantized and encoded to obtain a residual encoding model, which is then sent to the decoding end. This application uses the decoding model from a previous acquisition time as a reference to generate transformation parameters, which are then used to indicate the data change trends and patterns between the acquisition times. Subsequently, a residual representation model corresponding to the acquisition time is generated according to the transformation parameters. The residual representation model is used to fit the actual dynamic change trend of the video frame, and to offset the temporal error to the greatest extent. This avoids the error from accumulating over time, thereby ensuring the accuracy and reliability of the entire video data processing process and improving the performance of subsequent 3D video reconstruction. Attached Figure Description

[0052] Figure 1 This is a flowchart of the three-dimensional video processing method provided in the embodiments of this application.

[0053] Figure 2This is a flowchart illustrating the process of performing quantization encoding on a keyframe representation model to obtain a keyframe encoding model, as provided in an embodiment of this application.

[0054] Figure 3 This is a flowchart of obtaining the quantization step size provided in the embodiments of this application.

[0055] Figure 4 This is a flowchart of generating transformation parameters based on a decoding model, provided in an embodiment of this application.

[0056] Figure 5 This is a flowchart of the residual representation model for generating and processing frame sequences based on transformation parameters, provided in an embodiment of this application.

[0057] Figure 6 This is a flowchart provided in this application embodiment of performing inter-frame transformation on the decoding model according to the transformation parameters to obtain a reference decoding model.

[0058] Figure 7 This is a flowchart of the residual representation model provided in the embodiments of this application.

[0059] Figure 8 This is a flowchart of the rate-distortion joint optimization process provided in the embodiments of this application.

[0060] Figure 9 This is a schematic diagram of the overall process of three-dimensional video processing at the encoding end provided in the embodiments of this application.

[0061] Figure 10 This is an optional flowchart of the three-dimensional video processing method provided in the embodiments of this application.

[0062] Figure 11 This is a flowchart of decoding the acquired keyframe encoding model to obtain the keyframe decoding model, provided in an embodiment of this application.

[0063] Figure 12 This is a structural block diagram of a three-dimensional video processing device provided in another embodiment of this application.

[0064] Figure 13 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] After acquiring multi-viewpoint video sequences simultaneously from different perspectives, immersive holographic video technology focuses on achieving high-fidelity reconstruction of 3D dynamic scenes and supporting photorealistic real-time rendering from any viewpoint. Immersive communication encompasses new media service types such as XR communication and holographic communication. Looking back over the past few decades, video encoding and decoding technologies and corresponding standards have been built and developed around planar video; however, this is no longer sufficient to meet the application needs of immersive holographic communication in the future 6G era. With the rapid growth of immersive holographic video data, it poses extremely severe challenges to storage space and network bandwidth. How to efficiently and quickly achieve immersive holographic video encoding has become a crucial technical problem that urgently needs to be solved.

[0069] 3D Gaussian Splatting (3DGS) surpasses earlier 3D scene representation technologies such as point clouds, meshes, and Neural Radiation Fields (NeRF) in terms of 3D scene reconstruction quality, reconstruction speed, interactive freedom, and rendering speed. It has become a widely used 3D scene representation method, especially in the reconstruction of immersive holographic videos.

[0070] In related technologies, 3D Gaussian splashing technology uses neural networks to model the rotation and translation changes of Gaussian points at adjacent time points, using the inter-frame prediction results as a reference for the next frame to perform 3D reconstruction. However, this approach suffers from the problem of continuous accumulation of temporal errors, which negatively impacts the reconstruction performance of 3D video. Furthermore, most 3D Gaussian reconstruction techniques in related technologies lack compression coding processes. On the one hand, this leads to increased transmission costs and processing time; on the other hand, the inability to jointly optimize the coding bitrate and distortion also adversely affects reconstruction performance.

[0071] Based on this, embodiments of this application provide a three-dimensional video processing method, apparatus, device, and storage medium. Using a decoding model from a previous acquisition moment as a reference, transformation parameters are generated based on this decoding model. These transformation parameters indicate the data change trends and patterns between the acquisition moments. Subsequently, a residual representation model corresponding to the acquisition moment is generated according to the transformation parameters. This residual representation model is used to fit the actual dynamic change trends of the video frames, maximizing the offsetting of temporal errors and preventing errors from accumulating over time. This ensures the accuracy and reliability of the entire video data processing process and improves the performance of subsequent three-dimensional video reconstruction.

[0072] This application provides a three-dimensional video processing method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the three-dimensional video processing method in this application embodiment is described.

[0073] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0074] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0075] The three-dimensional video processing method provided in this application relates to the field of computer vision technology. This method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client that supports three-dimensional video processing, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The three-dimensional video processing method can be executed by the terminal or the server, or by the terminal and the server working together.

[0076] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device uses the 3D video processing method of this embodiment to provide related services and enhance the driving experience. The server can be an independent server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or network communication methods; this embodiment does not impose any limitations.

[0077] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0078] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0079] The following describes a three-dimensional video processing method in an embodiment of this application.

[0080] In one embodiment, the three-dimensional video processing method involved in this application needs to be completed collaboratively by the encoding end and the decoding end.

[0081] On the one hand, the encoding and decoding ends can be deployed within the same processing device. If the processing device has limited resources, performing encoding and decoding operations on the same device, and using encoding operations to compress and integrate the data, can effectively reduce the amount of data to be processed, lighten the burden on the device, and allow the device to complete subsequent processing tasks with limited resources. Furthermore, if there is a need to complete the reconstruction process quickly, such as in applications with high real-time requirements, performing encoding and decoding operations on the same device can reduce the time spent on intermediate steps such as data transmission, quickly achieving the conversion from raw data to the final reconstructed result, thereby ensuring processing efficiency.

[0082] On the other hand, the encoding and decoding ends can also be deployed in a distributed manner. Encoding is performed first at the encoding end, followed by transmission of the corresponding encoded bitstream, and then decoding is performed on the corresponding processing device at the decoding end. When transmission bandwidth is limited, transmitting the encoded bitstream can effectively reduce the amount of data transmitted and improve transmission efficiency. In this way, the entire reconstruction process will not be affected by delays in the transmission process, thereby improving the overall reconstruction efficiency.

[0083] It is understood that this embodiment does not impose specific restrictions on the deployment of the encoding and decoding ends. The processing flow of the encoding end and the decoding end will be described in detail below.

[0084] In one embodiment, Figure 1 This is an optional flowchart of the three-dimensional video processing method provided in the embodiments of this application. Figure 1 The method described herein, when applied to the encoding end, may include, but is not limited to, steps 110 to 130. It is also understood that this embodiment... Figure 1 The order of steps 110 to 130 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0085] Step 110: At multiple acquisition times, acquire video frames of the target scene from different perspectives. Use the video frames at the initial acquisition time as the keyframe sequence and the video frames at other acquisition times as the non-keyframe sequence.

[0086] In one embodiment, the target scene refers to those scenes that require 3D modeling. For the target scene, a 360-degree panoramic surround layout can be used to deploy image acquisition devices, enabling synchronous acquisition. This surround layout has the characteristic of omnidirectional coverage of the target scene, ensuring that no information from any angle is missed, thus allowing data to be collected from all directions. Different image acquisition devices correspond to different viewpoints, just like observing the target scene from different positions; the images seen from different positions will differ.

[0087] In one embodiment, images are captured from different perspectives at multiple acquisition times, so that a video can be obtained from each perspective. This video includes video frames corresponding to the respective acquisition time, and these video frames are the basic units that constitute the video. These frames are arranged sequentially according to the chronological order of the acquisition times, thus forming a continuous video.

[0088] In one embodiment, since all video frames except the initial video frame have a temporal reference relationship (meaning there is a motion correlation between the current video frame and the video frame corresponding to the previous acquisition time), this embodiment marks the initial acquisition time as the keyframe acquisition time and the other acquisition times as non-keyframe acquisition times. Accordingly, the video frames corresponding to different viewpoints at the initial acquisition time are considered as keyframe sequences, and the video frames corresponding to different viewpoints at other acquisition times are considered as non-keyframe sequences.

[0089] Step 120: Generate a keyframe representation model based on the keyframe sequence, perform quantization encoding on the keyframe representation model to obtain a keyframe encoding model, and decode the keyframe encoding model to obtain a keyframe decoding model.

[0090] In one embodiment, for a keyframe sequence, the Colmap calibration method in 3DGS can be used to perform camera calibration on video frames from different perspectives corresponding to the keyframe sequence, obtaining a sparse point cloud representing the target scene, as well as the intrinsic and extrinsic parameters corresponding to each image acquisition device. The sparse point cloud is then initialized, representing it as a keyframe representation model composed of multiple three-dimensional Gaussian distributions. Each three-dimensional Gaussian distribution can be considered as a three-dimensional ellipsoid, and multiple three-dimensional ellipsoids are used to fit the sparse point cloud.

[0091] Specifically, in this embodiment, the Colmap calibration method is used to analyze the feature information contained in each video frame. This feature information includes corner points, edges, and other significant visual features in the video frame. Then, a feature extraction algorithm is used to accurately identify these feature points. Next, a matching operation is performed based on the viewpoint correspondence between these feature points in different video frames. Finally, using the successfully matched feature point information, a sparse point cloud representing the target scene is obtained. This sparse point cloud is a discretized representation of the target scene in three-dimensional space, composed of numerous feature points. These feature points are relatively sparsely distributed and outline the approximate contours of the main objects in the target scene, as well as key features such as their spatial relationships.

[0092] Once the sparse point cloud is acquired, it is transformed into a form composed of multiple different 3D Gaussian distributions. Each 3D Gaussian distribution can be viewed as a 3D ellipsoid. Intuitively, a 3D ellipsoid has parameters such as its center position and semi-major and minor axes, and can cover a certain spatial region in a relatively flexible way that fits the actual spatial distribution pattern. Using multiple such 3D ellipsoids to fit the sparse point cloud means making the distribution and shape of these 3D ellipsoids as close as possible to the spatial form of the actual objects represented by the sparse point cloud. For example, in a target scene containing multiple irregularly shaped objects, by adjusting the center position, covariance, and other parameters of each 3D ellipsoid, it can be made to well wrap the sparse point cloud portion corresponding to the objects. Then, by combining multiple 3D ellipsoids, the spatial structure of the entire target scene can be accurately simulated.

[0093] In addition, after feature matching is completed, the intrinsic and extrinsic parameters corresponding to each image acquisition device are calculated. The intrinsic parameters reflect the camera's internal imaging characteristics, such as its focal length, principal point position, and potential lens distortion parameters; the extrinsic parameters reflect the camera's position and orientation in space, such as its rotation angle and translation vector relative to the target scene. These intrinsic and extrinsic parameters are used in subsequent rendering and reconstruction processes.

[0094] It is understandable that the keyframe representation model generated by 3DGS is only for illustration. The keyframe representation model can also be generated by Neural Radiation Field (NeRF), or it can be a point cloud model, voxel mesh model, etc.

[0095] In one embodiment, after obtaining the keyframe representation model, the encoding stage is required. (Refer to...) Figure 2 , Figure 2 This is a flowchart of the process of performing quantization encoding on a keyframe representation model to obtain a keyframe encoding model, as provided in this embodiment of the application. The process specifically includes the following steps:

[0096] Step 210: Obtain the quantization step size, and perform quantization operation on the first model parameters of the keyframe representation model according to the quantization step size to obtain the quantized representation model.

[0097] In one embodiment, reference is made to Figure 3 , Figure 3 This is a flowchart of obtaining the quantization step size provided in an embodiment of this application, which specifically includes the following steps:

[0098] Step 310: Obtain the first center position of multiple three-dimensional Gaussian distributions in the keyframe representation model.

[0099] In one embodiment, the keyframe representation model comprises multiple three-dimensional Gaussian distributions, each possessing at least a center position, a covariance matrix, a color parameter, and an opacity parameter. The center position determines the core of the three-dimensional Gaussian distribution in three-dimensional space, acting like a "locator point" and clearly defining the approximate coordinates of the spatial region primarily described by the distribution. The covariance matrix reflects the data dispersion of the Gaussian distribution across various dimensions in three-dimensional space and the correlation between different dimensions, similar to the size and rotation information of the Gaussian ellipsoid corresponding to the distribution. The color parameter determines the color appearance of the region corresponding to the distribution in visualization and other application scenarios, and is fitted using a spherical harmonic function. The opacity parameter reflects the transparency of the region. In this embodiment, the first center position is constructed by obtaining the center positions of all three-dimensional Gaussian distributions in the keyframe representation model.

[0100] Step 320: Input the first center position into the hash list to obtain the first context feature, and input the first context feature into the quantization prediction model for data processing to obtain the quantization step size.

[0101] In one embodiment, a binary hash table is pre-constructed to store the first contextual features corresponding to different locations. For example, the first contextual features can characterize the distribution of objects in the target scene, their spatial relationships with adjacent feature points, and any semantic information involved. By pre-constructing this binary hash table, this rich and diverse contextual information can be stored in a structured form, facilitating subsequent retrieval and use. Furthermore, this binary hash table can be optimized during application. The storage method, association rules, and related parameters in the hash table are continuously adjusted based on the processing frame sequence. In this embodiment, the first center position is input into the binary hash table, and a mapping operation is performed on its coordinate values ​​to obtain the corresponding first contextual features. The position information, as input, acts as an "index" for the binary hash table, thus clarifying the information to be searched.

[0102] Next, the first contextual features are input into the quantization prediction model to perform data processing and obtain the quantization step size. A multilayer perceptron can be used as the quantization prediction model here. After data prediction by this quantization prediction model, the quantization step size is obtained. Based on the obtained quantization step size, the first model parameters of the keyframe representation model are quantized to finally obtain the quantized representation model.

[0103] In one embodiment, the first model parameters of the keyframe representation model include, as mentioned above, the center positions of the three-dimensional Gaussian distributions, covariance matrices, color parameters, and opacity parameters. Quantization involves discretizing these originally continuously valued first model parameters according to the obtained quantization step size, transforming them into data that takes values ​​at specific intervals within a specific range. For example, the color parameter, which originally could take values ​​within a relatively wide range of continuous color values, is limited to several discrete color value points through quantization, based on the quantization step size. This approach helps reduce the amount of data, simplify the model representation, and retains key model features to some extent. Finally, after such quantization, a quantized representation model is obtained. Compared to the keyframe representation model, the quantized representation model is more compact and concise in terms of data format. The embodiments of this application do not limit the specific quantization calculation process.

[0104] Step 220: Perform entropy coding on the quantization representation model to obtain the keyframe coding model.

[0105] In one embodiment, entropy coding has many coding methods, such as arithmetic coding, Huffman tree coding, etc. This application embodiment does not limit the specific method of entropy coding.

[0106] Taking arithmetic coding as an example, it is an entropy coding technique based on probability statistics that treats the data of the entire quantization representation model as a whole sequence of symbols. When performing arithmetic coding on the quantization representation model, the probability distribution of each parameter and its value is first analyzed. For example, for different color parameter values ​​in the quantization representation model, their frequency of occurrence in the entire model is statistically analyzed to determine their respective coding intervals. Then, based on this probability information, all data in the quantization representation model is gradually mapped to the corresponding coding intervals. By continuously subdividing and determining the coding intervals, a unique keyframe coding model is finally generated. Compared to the quantization representation model, the keyframe coding model further reduces the amount of data, occupies less storage space, and presents the information in the original quantization representation model in a more compact and efficient coding form.

[0107] In one embodiment, after obtaining the keyframe encoding model, a related decoding operation is required at the encoding end. The keyframe encoding model is entropy decoded according to the entropy decoding method corresponding to entropy encoding to obtain a quantization decoding model. Similarly, following the inverse quantization process, the parameters of the quantization decoding model are inversely quantized according to the quantization step size to obtain the keyframe decoding model. After obtaining the keyframe decoding model, it is stored in a cache for subsequent use.

[0108] Step 130: Select non-keyframe sequences one by one as processing frame sequences. Obtain reference frame sequences based on the previous acquisition time of the processing frame sequences. Generate transformation parameters based on the decoding model corresponding to the reference frame sequences. Perform quantization and encoding operations on the transformation parameters to obtain encoded transformation parameters. Send the encoded transformation parameters to the decoding end. If residual identifiers exist, generate residual representation models corresponding to the processing frame sequences based on the transformation parameters. Perform quantization and encoding operations on the residual representation models to obtain residual encoded models. Send the residual encoded models to the decoding end. The decoding model corresponding to the first processing frame sequence is the keyframe decoding model.

[0109] In one embodiment, after processing the keyframe sequence, processing of the non-keyframe sequences begins. Since the non-keyframe sequences contain temporal information, they are selected one by one as processing frame sequences, and the frame sequence corresponding to the previous acquisition time of each processing frame sequence is used as the reference frame sequence. This allows for more effective utilization of the temporal correlation between non-keyframe sequences and between them and the keyframe sequences. Specifically, if the first non-keyframe sequence is selected, its corresponding reference frame sequence is the keyframe sequence; for other non-keyframe sequences, their corresponding reference frame sequence is the non-keyframe sequence corresponding to the previous acquisition time.

[0110] It is understood that the decoding model corresponding to the reference frame sequence can be obtained by the decoder sending the decoded data back to the encoder for storage, or by the encoder decoding the data using the same decoding method as the decoder. This embodiment does not limit this. The following explanation uses the first non-key frame sequence as the processing frame sequence as an example.

[0111] In one embodiment, a reference frame sequence corresponding to the processed frame sequence is first obtained, and then the decoding model corresponding to the reference frame sequence is obtained from the decoding end. For the first processed frame sequence, its decoding model is the keyframe decoding model corresponding to the keyframe encoding model. After obtaining the decoding model, data processing is performed on the processed frame sequence based on the corresponding decoding model.

[0112] In one embodiment, reference is made to Figure 4 , Figure 4 This is a flowchart of generating transform parameters based on a decoding model, provided in an embodiment of this application. The flowchart specifically includes the following steps:

[0113] Step 410: Obtain the second center position of multiple three-dimensional Gaussian distributions in the decoding model.

[0114] In one embodiment, the decoding model also includes multiple three-dimensional Gaussian distributions, each of which possesses at least a center position, a covariance matrix, a color parameter, and an opacity parameter. The center position of each of the three-dimensional Gaussian distributions in the decoding model is obtained to determine the second center position.

[0115] Step 420: Input the second center position into the hash list to obtain the second context features, and input the second context features into the motion transformation prediction model for data processing to obtain the transformation parameters.

[0116] In one embodiment, referring to step 320, the second center position is first input into a hash list to obtain the second contextual feature. Then, the second contextual feature is input into a motion transformation prediction model for data processing to obtain transformation parameters. The motion transformation prediction model can also be constructed using a multilayer perceptron.

[0117] Given that the parameters of a 3D Gaussian distribution often change due to object motion, scene lighting variations, or other dynamic factors at adjacent acquisition times, this embodiment utilizes a motion transformation prediction model to analyze the input second contextual features. This accurately captures the changes in these parameters over time and represents these changes as transformation parameters. Therefore, the transformation parameters indicate the amount of change in the predicted parameters of the second model parameters corresponding to the 3D Gaussian distribution in the decoding model at the acquisition time corresponding to the processing frame sequence. For example, for the center position of a certain 3D Gaussian distribution, the transformation parameters clearly indicate how much its coordinate position in 3D space has changed from the reference frame sequence to the acquisition time corresponding to the processing frame sequence.

[0118] Next, in one embodiment, refer to Figure 5 , Figure 5 This is a flowchart of the residual representation model for generating processing frame sequences based on transform parameters, provided in an embodiment of this application. The flowchart specifically includes the following steps:

[0119] Step 510: Perform inter-frame transformation on the decoding model according to the transformation parameters to obtain the reference decoding model.

[0120] In one embodiment, reference is made to Figure 6 , Figure 6 This is a flowchart provided in this application embodiment of performing inter-frame transformation on the decoding model according to the transformation parameters to obtain a reference decoding model, specifically including the following steps:

[0121] Step 610: Obtain at least the center position parameter, covariance parameter, color parameter, and opacity parameter for each three-dimensional Gaussian distribution of the decoding model.

[0122] Step 620: Based on the transformation parameters, obtain the center position update parameters corresponding to the center position parameter, the covariance update parameters corresponding to the covariance parameter, the color update parameters corresponding to the color parameter, and the opacity update parameters corresponding to the opacity parameter at the acquisition time corresponding to the processing frame sequence.

[0123] In one embodiment, based on the change in the predicted parameters at the acquisition time corresponding to the processing frame sequence indicated by the transformation parameters, the center position parameter is updated by adding the corresponding change to the center position parameter, the covariance parameter is updated by adding the corresponding change to the covariance parameter, the color parameter is updated by adding the corresponding change to the color parameter, and the opacity parameter is updated by adding the corresponding change to the opacity parameter.

[0124] Step 630: Update the three-dimensional Gaussian distribution based on the center position update parameter, covariance update parameter, color update parameter, and opacity update parameter to obtain the reference decoding model.

[0125] In one embodiment, the three-dimensional Gaussian distribution is updated based on the center position update parameter, covariance update parameter, color update parameter, and opacity update parameter to obtain a reference decoding model, which is then sent to the decoding end. It can be understood that the reference decoding model consists of multiple three-dimensional Gaussian distributions corresponding to the predicted processing frame sequence.

[0126] In one embodiment, a reference rendered image is obtained by rendering according to the reference decoding model. The reference rendered image is compared with the corresponding video frame for distortion. If the distortion level is within an acceptable threshold, there is no need to generate a residual identifier. Otherwise, a residual identifier is generated for the encoding end so that the encoding end can update the changed parameters according to the residual identifier, and then update the reference decoding model so that the rendering effect meets the actual requirements.

[0127] Step 520: Generate residual identifiers based on the reference decoding model.

[0128] In one embodiment, the specific process of generating a residual identifier based on a reference decoding model includes: rendering the reference decoding model to obtain a reference rendered image corresponding to the processed frame sequence; generating a residual identifier based on the distortion parameters between the reference rendered image and the video frames corresponding to the processed frame sequence; and sending the residual identifier to the encoding end when the residual identifier indicates that the transform parameters need to be updated.

[0129] In one embodiment, firstly, a rendering operation is performed on the reference decoding model to obtain the reference rendered image corresponding to the processed frame sequence. Taking 3DGS as an example, the rendering process is described as follows: at each viewpoint, the three-dimensional Gaussian distribution in the reference decoding model is first projected onto a two-dimensional space to obtain a two-dimensional Gaussian distribution. Then, the rendering work is carried out based on the two-dimensional Gaussian distribution to obtain the reference rendered image corresponding to each viewpoint.

[0130] Subsequently, residual identifiers are generated based on the distortion parameters between the reference rendered image and the corresponding video frames in the processed frame sequence. Specifically, the pixel differences between the reference rendered image from different viewpoints and the corresponding video frames are compared. Regions with pixel distortion levels greater than a preset threshold are identified as residual representation regions. Finally, all residual representation regions are aggregated to generate a residual identifier used to indicate the update of transformation parameters, and this residual identifier is sent to the encoding end. If the pixel distortion level is less than the preset threshold, no residual identifier is generated.

[0131] Step 530: Generate a residual representation model based on the residual identifier.

[0132] In one embodiment, the presence of residual identifiers indicates that a residual representation model needs to be generated. (See also...) Figure 7 , Figure 7 This is a flowchart of the process for generating a residual representation model provided in an embodiment of this application, which specifically includes the following steps:

[0133] Step 710: Determine the residual representation region of the processed frame sequence based on the residual identifier.

[0134] In one embodiment, the decoding end can perform region-by-region comparison between the reference rendered image and its corresponding video frame to analyze the differences in image presentation between the two. For example, comparisons can be made regarding color deviations, the clarity of object outlines, and the consistency of the layout of elements in the scene, thereby accurately identifying areas with poor rendering effects.

[0135] Subsequently, areas with poor rendering results are designated as residual representation areas. Residual representation areas indicate that, under the current encoding and rendering mechanisms, the image in that area significantly differs from the expected final result.

[0136] Step 720: Generate at least one three-dimensional Gaussian distribution for the residual representation region to form a residual representation model.

[0137] In one embodiment, the encoding end needs to add some three-dimensional Gaussian distributions to the residual representation region or segment the three-dimensional Gaussian distributions.

[0138] For example, in a region containing objects with complex textures, the original 3D Gaussian distribution might not accurately represent the texture details, while adding a new 3D Gaussian distribution can better capture this detailed information. Alternatively, while some large 3D Gaussian distributions can cover the corresponding area, their relatively "coarse" coverage makes it difficult to accurately fit the complex shapes and spatial layouts of objects within the area. Therefore, dividing it into several smaller 3D Gaussian distributions allows each smaller distribution to more specifically describe local spatial features, better matching the actual shape of the target scene in that region. This results in a more accurate and refined fit to the entire target scene, further improving the quality of subsequent rendering.

[0139] At this point, a residual representation model based on the residual representation region is obtained, which is also composed of multiple three-dimensional Gaussian distributions.

[0140] Understandably, the above-mentioned parameter update process can be executed multiple times until no more residual identifiers are received. For example, based on gradient judgment, if a large gradient consistently exists in the region, or if the gradient exceeds a certain threshold, a new 3D Gaussian distribution can be generated through random initialization or splitting from existing Gaussian distributions. This random initialization can also be achieved by re-annotating the processed frame sequence using COLMAP. A sparse point cloud is obtained using COLMAP; the location of this point cloud is used as the location of the Gaussian point, while other parameters are randomly initialized.

[0141] In addition, to further improve the reconstruction accuracy, the above quantization encoding process can be performed on only one or more of the transformation parameters and residual representation models. Similarly, the entropy decoding and dequantization processes at the decoding end are set accordingly.

[0142] In addition, the embodiments of this application also perform rate-distortion joint optimization processes on both key frame sequences and non-key frame sequences, and jointly optimize the transformation parameters at the encoding end, the update process of the residual representation model, and the decoding model at the decoding end, which can prevent the accumulation of temporal errors, improve prediction performance, and thus improve video coding efficiency.

[0143] In one embodiment, reference is made to Figure 8 , Figure 8 This is a flowchart of the rate-distortion joint optimization process provided in the embodiments of this application, which specifically includes the following steps:

[0144] Step 810: Obtain the rendering distortion loss term corresponding to the keyframe decoding model or the non-keyframe decoding model.

[0145] In one embodiment, the decoding model includes a keyframe decoding model or a non-keyframe decoding model. In this case, for the decoding model, the structural loss value of the rendered image and video frame is calculated using calculation methods such as D-SSIM, PSNR, and LPIPS as the rendering distortion loss term D.

[0146] Step 820: Obtain the bitrate loss term.

[0147] In one embodiment, the rate loss term is calculated by the decoding model or residual representation model during the quantization encoding process. For example, if the required code rate for encoding according to the quantized parameters is R2, then the rate loss term is R2, and the code rate can be calculated based on the information entropy.

[0148] Step 830: Calculate the rate-distortion joint loss term based on the rendering distortion loss term and the bitrate loss term.

[0149] In one embodiment, the rate-distortion joint loss term is used to optimize at least the decoding model, the residual representation model, and the transform parameters, where the rate-distortion joint loss term L is expressed as:

[0150] L=D+λR

[0151] Where λ represents the equilibrium factor, which is determined according to actual needs.

[0152] Through the above process, for keyframe sequences, the encoder generates a keyframe encoding model and sends it to the decoder. For non-keyframe sequences, if no residual identifier is received, it means that the rendering effect based on the current encoding transform parameters is satisfactory. Only the transform parameters need to be quantized and encoded according to the keyframe encoding model generation method to obtain the encoded transform parameters, which are then sent to the decoder for subsequent rendering. If a residual identifier is received, a residual representation model is generated, and the residual representation model is quantized and encoded using the same method to obtain the residual encoding model. Simultaneously, the encoding transform parameters are updated, and both the updated encoding transform parameters and the residual encoding model are sent to the decoder.

[0153] As can be seen from the above process, referring to Figure 9 , Figure 9 This is a schematic diagram of the overall process of three-dimensional video processing at the encoding end provided in the embodiments of this application.

[0154] First, based on the acquisition time, the video from multiple perspectives is divided into key frame sequences and non-key frame sequences. The purpose is to eliminate temporal redundancy in non-key frames, thereby significantly improving the compression efficiency of immersive holographic video.

[0155] For the keyframe sequence at the initial acquisition moment, keyframe rate-distortion optimization is performed, followed by obtaining a compact keyframe representation model at the encoding end. Next, quantization and entropy encoding operations are performed on the keyframe representation model to obtain the keyframe encoded model, which is then stored or transmitted as a binary bitstream. At the decoding end, entropy decoding and dequantization processing are performed on the keyframe encoded model to obtain the keyframe decoded model, which is then stored in the decoding buffer for interaction with the encoding end and for rendering operations at the decoding end.

[0156] Next, for the non-keyframe sequence in the multi-view video that is not at the initial acquisition time, the decoding model of the previous acquisition time needs to be extracted from the decoding buffer at the encoding end and used as a temporal reference. Then, non-keyframe rate-distortion optimization is performed on the non-keyframe sequence at the current acquisition time, thereby obtaining the residual representation model and transform parameters. Afterwards, quantization and entropy coding operations are performed on the residual representation model and transform parameters respectively, thus obtaining the residual coding model and coding transform parameters corresponding to the non-keyframe sequence.

[0157] Then, at the decoding end, entropy decoding and dequantization are sequentially performed on the residual coding model and coding transform parameters to obtain the decoding transform parameters and the residual decoding model. Based on this, inter-frame transform operations are performed on the temporal reference decoding model using the decoding transform parameters to obtain the reference decoding model. Furthermore, a residual fusion operation is performed on the reference decoding model and the residual decoding model to finally obtain the non-keyframe decoding model, which is then stored in the decoding buffer as the decoding model for the reference frame sequence of the next acquisition time. This process is repeated until all acquisition times have been processed.

[0158] The following is a detailed description of the processing flow at the decoding end.

[0159] In one embodiment, Figure 10 This is an optional flowchart of the three-dimensional video processing method provided in the embodiments of this application. Figure 10 The method described herein, when applied to the encoding end, may include, but is not limited to, steps 1010 to 1020. It is also understood that this embodiment... Figure 10 The order of steps 1010 to 1020 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0160] Step 1010: Decode the acquired keyframe encoding model to obtain the keyframe decoding model.

[0161] In one embodiment, reference is made to Figure 11 , Figure 11This is a flowchart of decoding the acquired keyframe encoding model to obtain a keyframe decoding model, provided in an embodiment of this application. The flowchart specifically includes the following steps:

[0162] Step 1110: Perform entropy decoding on the keyframe encoding model to obtain the quantization decoding model.

[0163] In one embodiment, the keyframe encoding model is entropy decoded according to the entropy decoding method corresponding to the entropy encoding at the encoding end to obtain the quantization decoding model. Further details are omitted here.

[0164] Step 1120: Obtain the quantization step size, and perform dequantization operation on the parameters of the quantization decoding model according to the quantization step size to obtain the keyframe decoding model.

[0165] In one embodiment, similarly, following the dequantization process of the encoding end quantization method, the parameters of the quantization decoding model are dequantized according to the quantization step size to obtain the keyframe decoding model.

[0166] Step 1020: For each processed frame sequence, obtain the corresponding encoding transformation parameters, obtain the decoding transformation parameters based on the encoding transformation parameters, perform inter-frame transformation using the decoding transformation parameters and the decoding model corresponding to the reference frame sequence to obtain the reference decoding model, use the reference decoding model as the non-key frame decoding model, if the processed frame sequence includes a residual encoding model, decode the residual encoding model to obtain the residual decoding model, perform residual fusion on the residual decoding model and the reference decoding model to obtain the non-key frame decoding model.

[0167] In one embodiment, the definition of the processing frame sequence is consistent with that of the encoder, so each processing frame sequence contains a corresponding reference frame sequence from the previous acquisition time. Since not all processing frame sequences require updating the transform parameters, not every processing frame sequence contains a residual coding model.

[0168] For a processed frame sequence containing only encoded transform parameters, the obtained encoded transform parameters are processed using the same entropy decoding and dequantization process as the keyframe encoding model described above, resulting in decoded transform parameters. These decoded transform parameters indicate the parameter changes between the current processed frame sequence and the reference frame sequence. Subsequently, similar inter-frame transform operations are performed using the decoded transform parameters and the decoding model corresponding to the reference frame sequence, yielding a reference decoding model. This reference decoding model is then used as the non-keyframe decoding model for this processed frame sequence. It is important to note that the decoding model for the first processed frame sequence is the keyframe decoding model.

[0169] For the processed frame sequence containing encoding transform parameters and residual encoding models, the obtained encoding transform parameters and residual encoding models are processed according to the same entropy decoding and dequantization process as the keyframe encoding model described above, thereby obtaining decoding transform parameters and residual decoding models. Next, a residual fusion operation is performed on the residual decoding model and the reference decoding model. This is done by adding a corresponding three-dimensional Gaussian distribution to the residual representation region using the residual decoding model, or by updating an existing three-dimensional Gaussian distribution, ultimately obtaining the corresponding non-keyframe decoding model.

[0170] As can be seen from the above process, a corresponding decoding model is generated at each acquisition moment, including a keyframe decoding model and at least one non-keyframe decoding model. In this process, the residual decoding model usually increases the number of three-dimensional Gaussian distributions. At this time, if the number of three-dimensional Gaussian distributions in a certain decoding model is too large, it can be pruned by judging the opacity threshold. This embodiment does not limit this.

[0171] After generating the corresponding decoding model at each acquisition moment, 3D reconstruction can be performed. Specifically, the multiple 3D Gaussian distributions contained in the keyframe decoding model and non-keyframe decoding model are projected in 2D onto each viewpoint to obtain the corresponding 2D Gaussian distribution. Image rendering is then performed on each 2D Gaussian distribution for the corresponding viewpoint to obtain the target rendered image for all viewpoints at each acquisition moment of the target scene. Finally, the 3D scene corresponding to the target scene is generated based on the target rendered image.

[0172] The technical solution provided in this application can be applied to immersive holographic video. By acquiring video frames of the target scene from different perspectives at multiple acquisition times, the video frames corresponding to the initial acquisition time are used as a keyframe sequence, and the video frames corresponding to other acquisition times are used as non-keyframe sequences. Then, a keyframe representation model is generated based on the keyframe sequence, and quantization encoding is performed on the keyframe representation model to obtain a keyframe encoding model. The keyframe encoding model is then decoded to obtain a keyframe decoding model. Next, non-keyframe sequences are selected one by one as processing frame sequences. A reference frame sequence is obtained based on the previous acquisition time of the processing frame sequence. Transformation parameters are generated based on the decoding model corresponding to the reference frame sequence, and quantization encoding is performed on the transformation parameters to obtain encoded transformation parameters, which are then sent to the decoding end. If a residual identifier exists, a residual representation model corresponding to the processing frame sequence is generated based on the transformation parameters, and quantization encoding is performed on the residual representation model to obtain a residual encoding model, which is then sent to the decoding end. This application uses the decoding model of a previous acquisition time as a reference to generate transformation parameters, using these transformation parameters to indicate the data change trends and patterns between previous and subsequent acquisition times. Subsequently, a residual representation model corresponding to the acquisition time is generated according to the transformation parameters. The residual representation model is used to fit the actual dynamic change trend of the video frame, and to offset the temporal error to the greatest extent. This avoids the error from accumulating over time, thereby ensuring the accuracy and reliability of the entire video data processing process and improving the performance of subsequent 3D video reconstruction.

[0173] This application also provides a three-dimensional video processing apparatus that can implement the above-described three-dimensional video processing method applied at the encoding end, see reference. Figure 12 The device includes:

[0174] Frame segmentation module 1210: used to acquire video frames of the target scene from different perspectives at multiple acquisition times, and to take the video frames corresponding to the initial acquisition time as the key frame sequence and the video frames corresponding to other acquisition times as the non-key frame sequence.

[0175] Keyframe encoding module 1220: Used to generate a keyframe representation model based on the keyframe sequence, perform quantization encoding on the keyframe representation model to obtain a keyframe encoding model, and decode the keyframe encoding model to obtain a keyframe decoding model.

[0176] Non-keyframe encoding module 1230: Used to select non-keyframe sequences one by one as processing frame sequences, obtain reference frame sequences based on the previous acquisition time of the processing frame sequences, generate transformation parameters based on the decoding model corresponding to the reference frame sequences, perform quantization encoding operations on the transformation parameters to obtain encoded transformation parameters, and send the encoded transformation parameters to the decoding end. If there is a residual identifier, generate a residual representation model corresponding to the processing frame sequences based on the transformation parameters, perform quantization encoding operations on the residual representation model to obtain a residual encoding model, and send the residual encoding model to the decoding end. The decoding model corresponding to the first processing frame sequence is the keyframe decoding model.

[0177] The specific implementation of the three-dimensional video processing device in this embodiment is basically the same as the specific implementation of the three-dimensional video processing method described above, and will not be repeated here.

[0178] This application also provides an electronic device, including:

[0179] At least one memory;

[0180] At least one processor;

[0181] At least one program;

[0182] The program is stored in a memory, and the processor executes the at least one program to implement the three-dimensional video processing method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0183] Please see Figure 13 , Figure 13 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0184] The processor 1301 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0185] The memory 1302 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1302 and is called and executed by the processor 1301 using the three-dimensional video processing method of the embodiments of this application.

[0186] The input / output interface 1303 is used to implement information input and output;

[0187] Communication interface 1304 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0188] Bus 1305 transmits information between various components of the device (e.g., processor 1301, memory 1302, input / output interface 1303, and communication interface 1304);

[0189] The processor 1301, memory 1302, input / output interface 1303 and communication interface 1304 are connected to each other within the device via bus 1305.

[0190] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described three-dimensional video processing method.

[0191] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0192] The three-dimensional video processing method, apparatus, device, and storage medium proposed in this application acquire video frames of the target scene from different perspectives at multiple acquisition times. The video frames at the initial acquisition time are used as a keyframe sequence, and the video frames at other acquisition times are used as non-keyframe sequences. A keyframe representation model is then generated based on the keyframe sequence. This keyframe representation model is quantized and encoded to obtain a keyframe encoding model. The keyframe encoding model is then decoded to obtain a keyframe decoding model. Next, non-keyframe sequences are selected one by one as processing frame sequences. A reference frame sequence is obtained based on the previous acquisition time of the processing frame sequence. Transformation parameters are generated based on the decoding model corresponding to the reference frame sequence. These transformation parameters are quantized and encoded to obtain encoded transformation parameters, which are then sent to the decoding end. If a residual identifier exists, a residual representation model corresponding to the processing frame sequence is generated based on the transformation parameters. This residual representation model is quantized and encoded to obtain a residual encoding model, which is then sent to the decoding end. This application uses the decoding model from a previous acquisition time as a reference to generate transformation parameters, which are then used to indicate the data change trends and patterns between the acquisition times. Subsequently, a residual representation model corresponding to the acquisition time is generated according to the transformation parameters. The residual representation model is used to fit the actual dynamic change trend of the video frame, and to offset the temporal error to the greatest extent. This avoids the error from accumulating over time, thereby ensuring the accuracy and reliability of the entire video data processing process and improving the performance of subsequent 3D video reconstruction.

[0193] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0194] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0195] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0197] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0198] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0200] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0201] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0202] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A three-dimensional video processing method, characterized in that, When applied to the encoding end, the method includes: At multiple acquisition times, video frames corresponding to the target scene from different perspectives are acquired. The video frames corresponding to the initial acquisition time are used as the key frame sequence, and the video frames corresponding to other acquisition times are used as non-key frame sequences. A keyframe representation model is generated based on the keyframe sequence. The keyframe representation model is then quantized and encoded to obtain a keyframe encoding model. Finally, the keyframe encoding model is decoded to obtain a keyframe decoding model. One by one, the non-key frame sequences are selected as processing frame sequences. A reference frame sequence is obtained based on the previous acquisition time of the processing frame sequence. Transformation parameters are generated based on the decoding model corresponding to the reference frame sequence. The transformation parameters are quantized and encoded to obtain encoded transformation parameters. The encoded transformation parameters are sent to the decoding end. If a residual identifier exists, a residual representation model corresponding to the processing frame sequence is generated based on the transformation parameters. The residual representation model is quantized and encoded to obtain a residual encoding model. The residual encoding model is sent to the decoding end. The decoding model corresponding to the first processing frame sequence is the key frame decoding model. The step of generating transformation parameters based on the decoding model corresponding to the reference frame sequence includes: obtaining the second center position of multiple three-dimensional Gaussian distributions in the decoding model; inputting the second center position into a hash list to obtain second context features; and inputting the second context features into a motion transformation prediction model for data processing to obtain the transformation parameters, wherein the transformation parameters include the change in the predicted parameters of the second model parameters corresponding to the three-dimensional Gaussian distributions in the decoding model at the acquisition time corresponding to the processed frame sequence; The step of generating the residual representation model corresponding to the processed frame sequence based on the transformation parameters includes: performing inter-frame transformation on the decoding model according to the transformation parameters to obtain a reference decoding model; generating the residual identifier according to the reference decoding model; and generating the residual representation model based on the residual identifier.

2. The three-dimensional video processing method according to claim 1, characterized in that, The step of performing quantization encoding on the keyframe representation model to obtain the keyframe encoding model includes: Obtain the quantization step size, and perform quantization operation on the first model parameters of the keyframe representation model according to the quantization step size to obtain the quantized representation model; Entropy coding is performed on the quantization representation model to obtain the keyframe coding model.

3. The three-dimensional video processing method according to claim 2, characterized in that, The acquisition of the quantization step size includes: Obtain the first center position of multiple three-dimensional Gaussian distributions in the keyframe representation model; The first center position is input into a hash list to obtain the first context feature, and the first context feature is input into a quantization prediction model for data processing to obtain the quantization step size.

4. The three-dimensional video processing method according to claim 1, characterized in that, The step of performing inter-frame transformation on the decoding model according to the transformation parameters to obtain a reference decoding model includes: At least the center position parameter, covariance parameter, color parameter, and opacity parameter corresponding to each three-dimensional Gaussian distribution of the decoding model are obtained; Based on the transformation parameters, the center position update parameters corresponding to the center position parameter, the covariance update parameters corresponding to the covariance parameter, the color update parameters corresponding to the color parameter, and the opacity update parameters corresponding to the opacity parameter are obtained at the acquisition time corresponding to the processing frame sequence. The reference decoding model is obtained by updating the three-dimensional Gaussian distribution based on the center position update parameter, the covariance update parameter, the color update parameter, and the opacity update parameter.

5. The three-dimensional video processing method according to claim 1, characterized in that, The step of generating the residual identifier based on the reference decoding model includes: The reference decoding model is rendered to obtain the reference rendered image corresponding to the processed frame sequence; The residual identifier is generated based on the distortion parameters between the reference rendered image and the video frames corresponding to the processed frame sequence.

6. The three-dimensional video processing method according to claim 1, characterized in that, The generation of the residual representation model includes: The residual representation region of the processed frame sequence is determined based on the residual identifier; Generate at least one three-dimensional Gaussian distribution for the residual representation region to form the residual representation model.

7. The three-dimensional video processing method according to claim 1, characterized in that, The method further includes: Obtain the rendering distortion loss term corresponding to the keyframe decoding model or the non-keyframe decoding model; Obtain the bitrate loss term, which is calculated by the decoding model or the residual representation model during the quantization encoding process; A rate-distortion joint loss term is calculated based on the rendering distortion loss term and the bitrate loss term, and the rate-distortion joint loss term is used to optimize at least the decoding model, the residual representation model, and the transform parameters.

8. A three-dimensional video processing method, characterized in that, When applied to the decoding end, the method includes: The acquired keyframe encoding model is decoded to obtain the keyframe decoding model; For each processed frame sequence, the corresponding encoding transformation parameters are obtained. A reference frame sequence is obtained based on the previous acquisition time of the processed frame sequence. Decoding transformation parameters are obtained based on the encoding transformation parameters. Inter-frame transformation is performed using the decoding transformation parameters and the decoding model corresponding to the reference frame sequence to obtain a reference decoding model. The reference decoding model is used as a non-keyframe decoding model. If the processed frame sequence includes a residual encoding model, the residual encoding model is decoded to obtain a residual decoding model. The residual decoding model and the reference decoding model are residually fused to obtain the non-keyframe decoding model. The decoding model of the first processed frame sequence is the keyframe decoding model. The encoding transformation parameters and the residual encoding model are generated and sent by the encoding end using the three-dimensional video processing method according to any one of claims 1 to 7.

9. The three-dimensional video processing method according to claim 8, characterized in that, Decoding the acquired keyframe encoding model to obtain a keyframe decoding model includes: Entropy decoding is performed on the keyframe encoding model to obtain a quantization decoding model; Obtain the quantization step size, and perform dequantization operation on the parameters of the quantization decoding model according to the quantization step size to obtain the keyframe decoding model.

10. The three-dimensional video processing method according to claim 8, characterized in that, The method further includes: The multiple three-dimensional Gaussian distributions contained in the keyframe decoding model and the non-keyframe decoding model are projected in two dimensions at each viewpoint to obtain the two-dimensional Gaussian distribution of the corresponding viewpoint. Each of the two-dimensional Gaussian distributions is rendered from the corresponding viewpoint to obtain the target rendered image corresponding to all viewpoints at each acquisition time of the target scene; A 3D scene corresponding to the target scene is generated based on the target rendered image.

11. A three-dimensional video processing device, characterized in that, Applied to the encoding end, the device includes: Frame segmentation module: used to acquire video frames of the target scene from different perspectives at multiple acquisition times, taking the video frames corresponding to the initial acquisition time as the key frame sequence and the video frames corresponding to other acquisition times as the non-key frame sequence. Keyframe encoding module: used to generate a keyframe representation model based on the keyframe sequence, perform quantization encoding on the keyframe representation model to obtain a keyframe encoding model, and decode the keyframe encoding model to obtain a keyframe decoding model; Non-keyframe encoding module: used to select the non-keyframe sequence one by one as the processing frame sequence, obtain the reference frame sequence according to the previous acquisition time of the processing frame sequence, generate transformation parameters according to the decoding model corresponding to the reference frame sequence, perform quantization encoding operation on the transformation parameters to obtain encoded transformation parameters, send the encoded transformation parameters to the decoding end, if there is a residual identifier, generate the residual representation model corresponding to the processing frame sequence based on the transformation parameters, perform quantization encoding operation on the residual representation model to obtain the residual encoding model, and send the residual encoding model to the decoding end. The decoding model corresponding to the first processing frame sequence is the keyframe decoding model. The step of generating transformation parameters based on the decoding model corresponding to the reference frame sequence includes: obtaining the second center position of multiple three-dimensional Gaussian distributions in the decoding model; inputting the second center position into a hash list to obtain second context features; and inputting the second context features into a motion transformation prediction model for data processing to obtain the transformation parameters, wherein the transformation parameters include the change in the predicted parameters of the second model parameters corresponding to the three-dimensional Gaussian distributions in the decoding model at the acquisition time corresponding to the processed frame sequence; The step of generating the residual representation model corresponding to the processed frame sequence based on the transformation parameters includes: performing inter-frame transformation on the decoding model according to the transformation parameters to obtain a reference decoding model; generating the residual identifier according to the reference decoding model; and generating the residual representation model based on the residual identifier.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the three-dimensional video processing method according to any one of claims 1 to 10.

13. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional video processing method according to any one of claims 1 to 10.

Images