Three-dimensional volume video coding and generating method, device, equipment and medium

Abstract

The invention discloses a three-dimensional volume video coding and generating method and device, electronic equipment and a storage medium, and relates to the technical field of volume videos, and the method comprises the steps: obtaining a key multi-view image in a multi-view video; reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model; encoding the preset transformation cache to obtain a code stream of a non-key scene, so as to obtain a non-key scene representation model in combination with the scene representation model at the previous moment after decoding; after traversing each non-critical multi-view image, a three-dimensional volume video is generated based on the critical scene representation model and each non-critical scene representation model. According to the method and the device, the effect of quickly and accurately generating the three-dimensional volume video with high quality is realized, the data processing efficiency in the volume video generation process is remarkably improved, and meanwhile, the data transmission quantity is reduced.

Description

Technical Field

[0001] The present application relates to the field of volumetric video technology, and in particular to a three-dimensional volumetric video encoding and generation method, device, electronic device and storage medium. Background Art

[0002] 3D volumetric video technology aims to reconstruct the structure of dynamic scenes to accurately present the dynamic changes of scenes and their spatial layout. It is widely used in fields such as virtual reality and augmented reality to provide users with a realistic immersive experience. Nowadays, with the rapid development of virtual reality and augmented reality applications, we need to quickly, accurately and high-quality reconstruct the 3D volumetric video of the scene from the input multi-viewpoint 2D images or videos to provide a good visual experience. However, the existing volumetric video generation methods usually need to model and compress scenes at different times, resulting in high computational complexity and slow scene reconstruction.

[0003] Therefore, how to efficiently encode and generate three-dimensional volumetric video is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the present application is to provide a three-dimensional volumetric video encoding and generation method, device, electronic device and storage medium, aiming to solve the technical problem of how to efficiently encode and generate three-dimensional volumetric video.

[0005] To achieve the above objectives, the present application proposes a three-dimensional volume video encoding and generation method, the three-dimensional volume video encoding and generation method comprising:

[0006] Acquire a multi-view video captured by multiple cameras, and determine a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video;

[0007] Reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video;

[0008] For any non-critical multi-view image, encode the preset transformation buffer to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain a non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model of the previous moment;

[0009] After traversing each non-critical multi-viewpoint image, a three-dimensional volume video is generated based on the critical scene representation model and each non-critical scene representation model.

[0010] In one embodiment, the step of encoding the key scene to obtain a code stream of the key scene includes:

[0011] Obtaining anchor point attributes of each anchor point in the key scene;

[0012] The attributes of each anchor point are quantized by using a preset scene compression model, and entropy coding is performed on the quantized attributes of each anchor point to obtain a code stream of the key scene.

[0013] In one embodiment, the three-dimensional volume video encoding and generation method further includes:

[0014] Calculating a first distortion loss and a first bit rate loss in the key scene reconstruction process, and constructing a first joint loss based on the first distortion loss and the first bit rate loss;

[0015] The anchor point attributes and the scene compression model are optimized according to the first joint loss to obtain adjusted anchor point attributes and a trained scene compression model, and based on the adjusted anchor point attributes and the trained scene compression model, the step of quantizing each anchor point attribute through the preset scene compression model is performed.

[0016] In one embodiment, the step of calculating the first distortion loss and the first bit rate loss in the key scene reconstruction process includes:

[0017] Calculating the first distortion loss according to the key multi-viewpoint image and the key scene representation model;

[0018] The first bit rate loss is calculated according to the probability distribution of the anchor point attribute, wherein the probability distribution is predicted by the preset scene compression model.

[0019] In one embodiment, the preset transformation cache includes a preset motion field, a preset feature residual field and a preset multi-layer perceptron MLP, and the step of encoding the preset transformation cache to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image for decoding and combining with the scene representation model at the previous moment to obtain the non-critical scene representation model of the multi-view video includes:

[0020] Encoding the preset motion field, the preset feature residual field and the preset multi-layer perceptron MLP to obtain a code stream of the non-critical scene;

[0021] When decoding the bitstream of the non-critical scene, obtaining the anchor point coordinates of each anchor point in the scene corresponding to the multi-viewpoint image at the previous moment;

[0022] Interpolate the decoded motion field according to the coordinates of each anchor point to obtain each motion representation feature vector, and input the motion representation feature vector into the decoded MLP to predict the offset change between the position offset of each Gaussian point at the current moment and the position offset of each Gaussian point in the scene representation model at the previous moment through the MLP;

[0023] Interpolate the decoded feature residual field according to the coordinates of each anchor point to obtain a feature vector representing each feature residual, and input the feature vector representing each feature residual into the decoded MLP to predict the feature residual between each current feature vector and each feature vector in the scene representation model at the previous moment through the MLP;

[0024] Each offset change is combined with the corresponding Gaussian point position offset in the scene representation model at the previous moment, and each feature residual is combined with the corresponding feature vector in the scene representation model at the previous moment to obtain the non-critical scene representation model.

[0025] In one embodiment, the three-dimensional volume video encoding and generation method further includes:

[0026] Calculating a second distortion loss and a second bit rate loss in the reconstruction process of the non-critical scene, and constructing a second joint loss based on the second distortion loss and the second bit rate loss;

[0027] Optimizing the preset motion field, the preset feature residual field, and the MLP according to the second joint loss to obtain an optimized motion field, an optimized feature residual field, and a trained MLP;

[0028] Based on the optimized motion field and the trained MLP, returning to the step of interpolating the decoded motion field according to the coordinates of each anchor point;

[0029] Based on the optimized feature residual field and the trained MLP, return to the step of interpolating the decoded feature residual field according to the coordinates of each anchor point.

[0030] In one embodiment, the step of calculating the second distortion loss and the second bit rate loss in the reconstruction process of the non-critical scene includes:

[0031] Calculating the second distortion loss according to the non-critical multi-viewpoint image and the non-critical scene representation model;

[0032] The second bit rate loss is calculated according to the number and frequency of binary numbers in the preset motion field and the preset characteristic residual field.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a three-dimensional volume video encoding and generation device, the three-dimensional volume video encoding and generation device comprising:

[0034] An acquisition module, used to acquire a multi-view video captured by multiple cameras, and determine a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video;

[0035] A first encoding module is used to reconstruct a key scene of the multi-view video according to the key multi-view image, and encode the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video;

[0036] The second key module is used to encode the preset transformation buffer for any non-critical multi-view image to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain a non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model of the previous moment;

[0037] A generation module is used to generate a three-dimensional volume video based on the key scene representation model and each non-key scene representation model after traversing each non-key multi-viewpoint image.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the three-dimensional volumetric video encoding and generation method as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the three-dimensional volume video encoding and generation method as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional volume video encoding and generation method as described above are implemented.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] The present application first obtains a multi-view video shot by multiple cameras, and determines the key multi-view image at the starting moment and the non-key multi-view images at subsequent moments in the multi-view video, so as to effectively identify and separate the key multi-view images that are crucial to scene reconstruction and the non-key multi-view images used to describe dynamic changes by capturing and distinguishing the key frames and the remaining frames in the video sequence, thereby providing an accurate time series data basis for subsequent scene reconstruction; reconstructs the key scene of the multi-view video according to the key multi-view images, encodes the key scene, and obtains a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video, constructs a three-dimensional scene model by processing the key multi-view images, and encodes it, thereby generating a compressed three-dimensional model code stream that can accurately represent the key scene, so that the original key scene can be quickly and with high quality restored during decoding, providing a key scene representation model for the entire video sequence. A stable reconstruction starting point; for any non-critical multi-view image, the preset transformation cache is encoded to obtain the code stream of the non-critical scene corresponding to the non-critical multi-view image, so as to be combined with the scene representation model of the previous moment after decoding to obtain the non-critical scene representation model of the multi-view video. By encoding only the transformation cache, the dynamic change part between frames is efficiently captured and encoded, so that a three-dimensional representation of the non-critical scene can be quickly generated through a small amount of parameter adjustment during decoding, which significantly improves the reconstruction speed and reduces the amount of transmitted data; after traversing each non-critical multi-view image, a three-dimensional volumetric video is generated based on the key scene representation model and each non-critical scene representation model. Through the three-dimensional video synthesis technology, the key scene representation model and the non-critical scene representation model are combined to generate a complete three-dimensional volumetric video, which can realistically reproduce the dynamic changes and spatial layout of the original scene, providing users with a high-quality immersive experience.

[0043] In summary, the present application realizes rapid reconstruction of non-critical scenes by combining a preset transformation cache with a scene representation model of the previous moment during the reconstruction process of non-critical scenes, thereby avoiding the high computational complexity and slow scene reconstruction speed caused by complete modeling and compression of the scene at each moment in traditional volumetric video generation methods, and achieves the effect of fast, accurate and high-quality generation of three-dimensional volumetric video, providing users with a realistic immersive experience, significantly improving the data processing efficiency of the volumetric video generation process, and reducing the amount of data transmission after encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of a flow chart provided for Embodiment 1 of the three-dimensional volumetric video encoding and generation method of the present application;

[0047] Figure 2 A first cache diagram of the three-dimensional volume video encoding and generation method provided in Embodiment 1 of the present application;

[0048] Figure 3 A second cache diagram of the three-dimensional volume video encoding and generation method provided in the first embodiment of the present application;

[0049] Figure 4 A schematic diagram of a flow chart provided for Embodiment 2 of the three-dimensional volumetric video encoding and generation method of the present application;

[0050] Figure 5 A hash grid diagram of a three-dimensional volume video encoding and generation method provided in Embodiment 2 of the present application;

[0051] Figure 6 A flowchart of the third embodiment of the three-dimensional volumetric video encoding and generation method of the present application is provided;

[0052] Figure 7 A schematic diagram of a simplified flow chart of a three-dimensional volume video encoding and generation method provided in Embodiment 3 of the present application;

[0053] Figure 8 A schematic diagram of a coding framework of a three-dimensional volume video coding and generation method provided in Embodiment 3 of the present application;

[0054] Fig. 9 This is a schematic diagram of the module structure of the three-dimensional volume video encoding and generation device according to an embodiment of the present application;

[0055] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the three-dimensional volume video encoding and generation method in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of the embodiment of the present application is: obtaining a multi-view video shot by multiple cameras, and determining a key multi-view image at a starting moment and non-key multi-view images at subsequent multiple moments in the multi-view video; reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video; for any non-key multi-view image, predicting the transformation anchor point parameters between the non-key scene corresponding to the non-key multi-view image and the scene corresponding to the multi-view image at the previous moment through a preset artificial neural network model, and encoding the transformation anchor point parameters to obtain a code stream of the non-key scene, so that after decoding, the non-key scene representation model of the multi-view video is obtained in combination with the scene representation model of the previous moment; after traversing each non-key multi-view image, a three-dimensional volumetric video is generated based on the key scene representation model and each non-key scene representation model.

[0060] With the rapid development of virtual reality and augmented reality applications, we need to quickly, accurately, and with high quality reconstruct the three-dimensional volumetric video of the scene from the input multi-viewpoint two-dimensional images or videos to provide a good visual experience. However, the existing volumetric video generation methods usually need to model and compress the scenes at different times, resulting in high computational complexity and slow scene reconstruction. Therefore, how to efficiently encode and generate three-dimensional volumetric videos is a problem that needs to be solved urgently.

[0061] The present application provides a solution, which realizes the rapid reconstruction of non-critical scenes by combining a preset transformation cache with a scene representation model of the previous moment during the reconstruction process of non-critical scenes, thereby avoiding the high computational complexity and slow scene reconstruction speed caused by complete modeling and compression of the scene at each moment in the traditional volumetric video generation method, and realizes the effect of fast, accurate and high-quality generation of three-dimensional volumetric video, providing users with a realistic immersive experience, significantly improving the data processing efficiency of the volumetric video generation process, and reducing the amount of data transmitted after encoding.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions. The following takes an electronic device as an example to illustrate this embodiment and the following embodiments.

[0063] Based on this, the embodiment of the present application provides a three-dimensional volume video encoding and generation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the 3D volume video encoding and generation method of the present application.

[0064] In this embodiment, the three-dimensional volume video encoding and generation method includes steps S10 to S40:

[0065] Step S10, acquiring a multi-view video captured by multiple cameras, and determining a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video;

[0066] It should be noted that multi-view video refers to video shot simultaneously by multiple cameras from multiple different angles, which can be used to reconstruct the three-dimensional information of the scene; key multi-view images refer to key frames in the multi-view video used to reconstruct the initial state of the three-dimensional scene, which contain rich scene information and details; non-key multi-view images refer to frames other than key frames, which are used to describe the dynamic changes of the scene over time.

[0067] It is understandable that, since the traditional volumetric video generation method needs to model and compress the scenes at different times, that is, the same processing is performed on different multi-view images, step S10 can avoid using the multi-view images at different times as the data basis for modeling, thereby causing complex calculation problems. By distinguishing the key image at the starting moment and the non-key images at subsequent moments, a reliable data basis is provided for subsequent different processing of different images, thereby reducing the operation of complex processing of non-key images.

[0068] Exemplarily, a multi-view video shot by multiple cameras is obtained, and the image at time t=0 in the video is determined as the key multi-view image, and the image at time t>0 (such as t=1, t=2, ...) in the video is determined as the non-key multi-view image.

[0069] Step S20, reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video;

[0070] It should be noted that the key scene refers to a static model of a three-dimensional scene reconstructed by key multi-viewpoint images, which represents a key state in a video sequence; the key scene bitstream refers to the compressed data stream obtained after encoding the key scene, which is used for transmission and storage; the key scene representation model refers to the model decoded from the key scene bitstream, which can represent the three-dimensional structure and appearance of the scene.

[0071] It is understandable that, since reconstruction of a scene requires at least the establishment of an accurate initial scene model, performing step S20 can avoid inaccuracies in subsequent scene reconstruction, thereby providing an effective reconstruction basic platform for achieving high-quality three-dimensional scene reconstruction.

[0072] Exemplarily, the Colmap method is used to calibrate the camera on the key multi-view images to obtain a sparse point cloud and the intrinsic and extrinsic parameters of each camera, and the coefficient point cloud is used to initialize the anchor-based Gaussian representation model. The anchor point properties are then continuously optimized according to the key multi-view images, and the obtained intrinsic and extrinsic parameters are combined to achieve the best reconstruction of the key scene.

[0073] In a feasible implementation manner, the step of encoding the key scene in step S20 to obtain the code stream of the key scene may include steps S21 to S22:

[0074] Step S21, obtaining the anchor point attributes of each anchor point in the key scene;

[0075] It should be noted that anchor points refer to reference points that represent specific positions or points in a three-dimensional scene. They are key points in scene reconstruction and are used to describe the structure of the scene. Anchor point attributes refer to state information related to the anchor point, including anchor point coordinates, feature vectors, sizes, and Gaussian point position offsets. These attributes help define the specific state of the anchor point in three-dimensional space.

[0076] Step S22, quantizing each anchor point attribute by using a preset scene compression model, and performing entropy coding on the quantized anchor point attributes to obtain a code stream of the key scene.

[0077] It should be noted that the preset scene compression model is a method of compressing and encoding a reconstructed scene, or compressing a scene while reconstructing the scene. It can be a context model, a hyper-prior model, etc., wherein the context model may include a hash grid and an MLP (Multilayer Perceptron). The MLP predicts the quantization step and probability distribution of each anchor point attribute based on the context information stored in the hash grid (such as the attributes of adjacent anchor points). The predicted quantization step is used to quantize each anchor point attribute, and the predicted probability distribution is used to calculate the bit rate loss in the scene reconstruction process. The quantization process of the anchor point attributes is optimized according to the context information of the scene, which can avoid quantization errors caused by not considering the context information, achieve more accurate anchor point attribute representation, improve the accuracy and efficiency of the quantization process, and thus improve the three-dimensional scene reconstruction; entropy coding is a data compression technology that encodes according to the probability of occurrence of each symbol in the data, uses short codewords for high-frequency symbols, and uses long codewords for low-frequency symbols, thereby reducing the redundancy of the overall data.

[0078] It can be understood that since the scene needs to be compressed and quantized during the scene transmission process, and the quantized anchor point attribute data usually still has redundancy, step S22 is performed to quantize and entropy encode the anchor point attributes to effectively compress the anchor point attribute data, reduce the amount of data required for storage and transmission, avoid the waste of bandwidth and storage resources caused by uncompressed data transmission, and achieve efficient data compression, thereby significantly reducing the data size and improving the efficiency of data transmission and storage, while maintaining the necessary information for reconstructing the scene.

[0079] Exemplarily, the quantization step size of each anchor point attribute is predicted according to the context model, and the anchor point attributes are quantized according to the quantization step size output by the context model, and then the quantized anchor point attributes are entropy encoded using an arithmetic encoder to obtain a binary code stream corresponding to the key scene.

[0080] In this implementation, by quantizing and entropy encoding the anchor point attributes, data redundancy problems are avoided during data transmission, and efficient compression and accurate representation of key scene anchor point attributes are achieved, thereby improving the efficiency of data transmission and storage while ensuring the quality of three-dimensional scene reconstruction.

[0081] Step S30, for any non-critical multi-view image, encode the preset transformation buffer to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain a non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model of the previous moment;

[0082] It should be noted that the preset transform cache is a data structure that stores at least binary transform parameters for encoding non-critical multi-view images, including motion fields and feature residual fields, and pre-configured multi-layer perceptrons (MLPs). These parameters can be used to describe the changes between adjacent frame images and the dynamic characteristics of the scene. For different 3D dynamic scenes, transform caches with different hash grid structures are configured. Please refer to Figure 4 , only the 3D grid structure A may be used, or the 2D grid structure B representing three planes may be used, or the 2D-3D grid composite structure C may be used. In this implementation, the hash grid structure of the transform cache is not specifically limited; the code stream of the non-critical scene refers to the compressed data stream obtained after encoding the transform cache of the non-critical scene; the non-critical scene representation model refers to the model obtained by combining the scene representation model of the previous moment and the code stream decoding of the current non-critical scene, which is used to represent the scene state at the non-critical moment.

[0083] In addition, it should be noted that the scene representation model at the previous moment may be a critical scene representation model or a non-critical scene representation model.

[0084] It can be understood that since the non-key multi-view images change less than the key frames, in order to efficiently describe the dynamic changes of the scene, step S30 is performed, which can avoid the computational burden of completely reconstructing each non-key frame, i.e., non-key multi-view image. By only encoding the preset transformation cache, the scene representation model of the previous moment is combined after decoding to obtain the scene representation model of the current moment, thereby reducing the encoding and decoding time and the required storage space, thereby achieving fast and accurate scene dynamic representation.

[0085] Exemplarily, for any non-critical multi-view image at time t, the preset motion field, preset feature residual field and preset MLP in the preset transformation cache are encoded to obtain a code stream of the non-critical scene, wherein the preset motion field and the preset feature residual field are stored in the transformation cache in the form of a binary hash table, and the binary hash table replaces the feature type from a 32-bit floating point number to a 1-bit binary number, which greatly reduces the storage occupancy of the transformation anchor parameter cache. After decoding the code stream, the decoded motion field and feature residual field are interpolated according to the anchor coordinates in the scene representation model at the previous moment, and the interpolation result is predicted by the decoded MLP to obtain the change of some anchor point attributes between the scene representation model at the current moment and the scene representation model at the previous moment, including the offset change and feature residual of each Gaussian point position, so as to obtain the scene representation model at the current moment according to the anchor point attributes of the scene representation model at the previous moment and the change of some anchor point attributes.

[0086] In a feasible implementation manner, the preset transformation cache includes a preset motion field, a preset feature residual field and a preset multi-layer perceptron MLP. The step of encoding the preset transformation cache in step S30 to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain the non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model at the previous moment, may include steps S31 to S35:

[0087] Step S31, encoding the preset motion field, the preset feature residual field and the preset multi-layer perceptron MLP to obtain a code stream of the non-critical scene;

[0088] It should be noted that the preset motion field refers to a predefined field that contains the motion patterns that may appear in the scene and is used to predict the position offset of the Gaussian point; the preset feature residual field refers to a predefined field that contains the change pattern of the feature vector in the scene and is used to predict the residual of the feature vector.

[0089] Step S32, when decoding the bitstream of the non-critical scene, obtaining the anchor point coordinates of each anchor point in the scene corresponding to the multi-viewpoint image at the previous moment;

[0090] It should be noted that the scene corresponding to the multi-viewpoint image at the previous moment can be a key scene or a non-key scene; the anchor point coordinates refer to the specific position coordinates of the anchor point in the three-dimensional space, which are the basic reference points for scene reconstruction.

[0091] Step S33, interpolating the decoded motion field according to the coordinates of each anchor point to obtain each motion characterization feature vector, and inputting the each motion characterization feature vector into the decoded MLP, so as to predict the offset change between the position offset of each Gaussian point at the current moment and the position offset of each Gaussian point in the scene representation model at the previous moment through the MLP;

[0092] It should be noted that the motion characterization feature vector refers to a vector used to describe the motion characteristics of the Gaussian point, and the vector contains information about the position offset of the Gaussian point.

[0093] It is understandable that, since it is necessary to predict the movement of Gaussian points in the scene, performing step S33 can avoid directly calculating the specific offset of all Gaussian points, thereby reducing the amount of calculation, achieving efficient prediction of the offset of Gaussian points in the scene, and improving the efficiency and accuracy of dynamic scene reconstruction.

[0094] Exemplarily, the preset motion field M in the preset transformation cache is transformed according to the anchor point coordinate x. t Interpolation is performed to obtain the motion characterization feature vector interp(x,M t ), and the motion characterization feature vector is input into the MLP to perform the offset change Δo of the Gaussian point position offset t predict:

[0095] Δo t =MLP1(interp(x,M t ))

[0096] The offset change Δo t Used to decode and combine the corresponding Gaussian point position offset o in the scene representation model at the previous moment t-1 , get the Gaussian point position offset o of the non-critical scene representation model at the current moment t , that is o t =o t-1 +Δo t .

[0097] Step S34, interpolating the decoded feature residual field according to the coordinates of each anchor point to obtain feature vectors representing each feature residual, and inputting the feature vectors representing each feature residual into the decoded MLP to predict the feature residuals between each current feature vector and each feature vector in the scene representation model at the previous moment through the MLP.

[0098] It should be noted that the feature residual representation feature vector refers to a vector used to describe the change characteristics of the feature vector, and the vector contains the information of the feature vector residual.

[0099] In addition, it should be noted that in this embodiment, the motion field and the feature residual field are modeled separately based on the transformation cache. Different training methods can be adopted for different three-dimensional dynamic scenes: please refer to Figure 2 , use a transform cache to simultaneously model the motion field and feature residual field, that is, interpolate the binary hash table A through the anchor point coordinate x to obtain the corresponding feature vector B, and input the feature vector into the MLP to predict the offset change Δo and feature residual Δf through the MLP, where Δo constitutes the motion field M and Δf constitutes the feature residual field F; use two or more transform caches to model motion and feature residuals respectively. For non-critical scenes, the transform cache model at the current moment is obtained after two or more stages of training. Please refer to Figure 3 In the first stage, we can only learn the transformation cache that characterizes the offset change of the Gaussian point. In the second stage, we fix the motion transformation cache parameters of the first stage and learn the transformation cache model that characterizes the feature residual. That is, we interpolate the binary hash table A through the anchor point coordinate x to obtain the corresponding feature vector B, and input the feature vector into the MLP to predict the offset change Δo or the feature residual Δf through the MLP, where Δo constitutes the motion field M and Δf constitutes the feature residual field F.

[0100] It is understandable that, since it is necessary to describe the changes in the features in the scene, performing step S33 can avoid complete reconstruction of each feature vector, reduce unnecessary data processing operations, thereby improving data processing efficiency while maintaining the details of scene reconstruction.

[0101] Exemplarily, the preset feature residual field F in the preset hash table is calculated according to the anchor point coordinate x. t Interpolation is performed to obtain the characteristic residual representation feature vector interp(x,F t ), and the feature residual represents the feature vector and inputs it into the MLP to perform the feature residual Δf of the feature vector t predict:

[0102] Δf t =MLP2(interp(x,F t ))

[0103] Among them, the characteristic residual Δf t Used to decode and combine the corresponding feature vector f in the scene representation model of the previous moment t-1 , get the feature vector f of the non-critical scene representation model at the current moment t , that is, f t=f t-1 +Δf t .

[0104] Step S35, combining each offset change with the corresponding Gaussian point position offset in the scene representation model at the previous moment, and combining each feature residual with the corresponding feature vector in the scene representation model at the previous moment, to obtain the non-critical scene representation model.

[0105] For example, each offset is changed by Δo t The offset o of the Gaussian point position corresponding to the scene representation model at the previous moment t-1 Combined, we can get the Gaussian point position offset o of the non-critical scene representation model at the current moment t , that is o t =o t-1 +Δo t ; The feature residual Δf t The feature vector f corresponding to the scene representation model at the previous moment t-1 Combine them to get the feature vector f of the non-critical scene representation model at the current moment t , that is, f t =f t-1 +Δf t Based on the Gauss point position offset o t , eigenvector f t , as well as the anchor point coordinates and scale of the previous moment, to obtain the non-critical scene representation model.

[0106] In this implementation, by performing anchor point coordinate positioning, interpolation operations, and neural network prediction, the high complexity and low efficiency problems caused by directly calculating the movement of all points and feature changes are avoided, and efficient reconstruction and accurate representation of dynamic scenes are achieved, ensuring the continuity and details of the scene. At the same time, the data processing process is optimized, reducing the computational burden and data volume.

[0107] Step S40: after traversing each non-critical multi-viewpoint image, generating a three-dimensional volumetric video based on the critical scene representation model and each non-critical scene representation model.

[0108] It can be understood that in order to synthesize a continuous 3D volumetric video, step S40 is performed, which provides a method for effectively generating a 3D volumetric video.

[0109] Exemplarily, after traversing each non-critical multi-viewpoint image, each non-critical scene representation model corresponding to each non-critical multi-viewpoint image is rendered from a virtual viewpoint. At the same time, the key scene representation model corresponding to the key multi-viewpoint image is also rendered from a virtual viewpoint to obtain the final desired three-dimensional volumetric video. This allows users to use a player with functions such as rotation, translation, scrolling, and zooming to independently select the viewing angle and playback time through operations such as dragging and clicking, thereby achieving free interaction of the three-dimensional volumetric video.

[0110] The present embodiment provides a three-dimensional volumetric video encoding and generation method. In the reconstruction process of non-critical scenes, a preset transformation cache is combined with a scene representation model of the previous moment to achieve rapid reconstruction of non-critical scenes, thereby avoiding the high computational complexity and slow scene reconstruction speed caused by complete modeling and compression of the scene at each moment in the traditional volumetric video generation method, achieving the effect of quickly, accurately and high-quality generation of three-dimensional volumetric video, providing users with a realistic immersive experience, significantly improving the data processing efficiency of the volumetric video generation process, and reducing the amount of data transmission after encoding.

[0111] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 5 The preset context model at least includes a hash grid and a multi-layer perceptron MLP, and the three-dimensional volume video encoding and generation method further includes steps S201 to S202:

[0112] Step S201, calculating a first distortion loss and a first bit rate loss in the key scene reconstruction process, and constructing a first joint loss based on the first distortion loss and the first bit rate loss;

[0113] It should be noted that the first distortion loss refers to the difference measure between the reconstructed scene and the original scene caused by quantization or encoding during the reconstruction of the key scene, which reflects the decline in reconstruction quality; the first bit rate loss refers to the amount of data consumed in the encoding process to transmit or store the reconstructed scene information, which reflects the loss of coding efficiency; the first joint loss is a comprehensive indicator that combines distortion loss and bit rate loss, and is used to balance the relationship between reconstruction quality and coding efficiency in the optimization process.

[0114] It is understandable that since traditional methods often fail to consider the storage occupancy of the model during scene reconstruction, and the larger model size poses a challenge to the transmission of volumetric video, affecting the practical application of volumetric video, step S201 is performed. By simultaneously considering the reconstruction quality (i.e., the first distortion loss) and the data transmission cost (i.e., the first bit rate loss), the problem of low availability of the volumetric video generation method due to the simple pursuit of reconstruction quality can be avoided, thereby providing an effective optimization supervision method for achieving faster and more stable generation of volumetric video without losing too much image quality.

[0115] Exemplarily, the first distortion loss D in the key scene reconstruction process is calculated (0) and the first rate loss R (0) , and the first joint loss L is calculated by the following formula (0) :

[0116] L (0) =D (0) +λR (0)

[0117] Wherein, λ represents the weight coefficient of the first code rate loss.

[0118] In a feasible implementation manner, the step of calculating the first distortion loss and the first bit rate loss in the key scene reconstruction process in step S201 may include steps S210 to S220:

[0119] Step S210, calculating the first distortion loss according to the key multi-viewpoint image and the key scene representation model;

[0120] Exemplarily, the first distortion loss D is calculated by the following formula: (0) :

[0121] D (0) =(1-λ ssim )×L1+λ ssim ×(1-SSIM)

[0122] Among them, λ ssim The weight used to weigh the L1 loss and SSIM loss, where L1 is the absolute error and SSIM (Structural Similarity Index Measure) is a parameter determined based on the similarity between the key multi-view images and the key scene representation model.

[0123] Step S220: Calculate the first bit rate loss according to the probability distribution of the anchor point attribute, wherein the probability distribution is predicted by the preset scene compression model.

[0124] Exemplarily, the first bit rate loss R is calculated by the following formula: (0) :

[0125]

[0126] Among them, s represents the anchor point attributes, including the feature vector f, size l and Gaussian point position offset o, is the quantized anchor point attribute, Represents the probability distribution of anchor attributes.

[0127] In this implementation, by calculating the first distortion loss and the first code rate loss, an effective data basis is provided for comprehensively considering the reconstruction quality and the data transmission cost.

[0128] Step S202: optimize the anchor point attributes and the scene compression model according to the first joint loss to obtain adjusted anchor point attributes and the trained scene compression model; and based on the adjusted anchor point attributes and the trained scene compression model, perform the step of quantizing each anchor point attribute by using the preset scene compression model.

[0129] It is understandable that since it is often difficult to directly achieve optimal performance through the process of quantizing and compressing anchor point attributes, step S202 is performed to improve the accuracy of the reconstruction algorithm, reduce reconstruction errors, and improve encoding efficiency by optimizing the anchor point attributes and the scene compression model, thereby achieving higher quality and more efficient three-dimensional volumetric video reconstruction.

[0130] Exemplarily, when the scene compression model is a context model (including a hash grid and an MLP), the calculated joint loss is minimized by continuously adjusting the attributes of each anchor point, the parameters in the hash grid in the context model, and the weights and biases of each layer of the neural network in the MLP. The anchor point attributes that minimize the joint loss, the parameters in the hash grid in the context model, and the weights and biases of each layer of the neural network in the MLP are used as the adjusted anchor point attributes and the trained context model, and the quantization step size and probability distribution of each adjusted anchor point attribute are predicted based on the trained context model.

[0131] In this embodiment, by performing joint loss optimization on the anchor point attributes and the scene compression model, the problems of reduced reconstruction quality and low data transmission efficiency that may be caused by optimizing distortion or bit rate alone during the three-dimensional volumetric video reconstruction are avoided. This ensures that the quality of the reconstructed scene is ensured while improving the data compression efficiency, thereby achieving a balance between high quality and efficient coding, and improving the performance of the overall video reconstruction system.

[0132] Based on the first embodiment and / or the third embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned embodiments 1 and 2 can be referred to the above introduction, and will not be repeated in the following. Figure 6 The three-dimensional volume video encoding and generation method further includes steps S301 to S304:

[0133] Step S301, calculating a second distortion loss and a second bit rate loss in the reconstruction process of the non-critical scene, and constructing a second joint loss based on the second distortion loss and the second bit rate loss;

[0134] It should be noted that the second distortion loss refers to the difference between the reconstructed scene and the original scene during the reconstruction of non-critical scenes, and this difference measures the loss of reconstruction quality; the second bit rate loss refers to the amount of data consumed for encoding non-critical scene information, and this amount of data reflects the efficiency loss of data transmission or storage during the encoding process; the second joint loss is a composite loss function that combines distortion loss and bit rate loss, which is used to balance reconstruction quality and coding efficiency during the optimization process.

[0135] It is understandable that since traditional methods often fail to consider the storage occupancy of the model during scene reconstruction, and the larger model size poses a challenge to the transmission of volumetric video, affecting the practical application of volumetric video, step S301 is performed. By simultaneously considering the reconstruction quality (i.e., the second distortion loss) and the data transmission cost (i.e., the second bit rate loss), it is possible to avoid the problem of low availability of the volumetric video generation method due to the simple pursuit of reconstruction quality, thereby further reducing the model size without losing too much image quality, and achieving faster and more stable generation of volumetric video, providing an effective optimization supervision method.

[0136] Exemplarily, the second distortion loss D in the reconstruction process of the non-critical scene is calculated as (t) and the second rate loss R (t) , and the second joint loss L is calculated by the following formula (t) :

[0137] L (t) =D (t) +λR (t)

[0138] Wherein, λ represents the weight coefficient of the second code rate loss.

[0139] In a feasible implementation manner, the step of calculating the second distortion loss and the second bit rate loss in the reconstruction process of the non-critical scene in step S301 may include steps S310 to S320:

[0140] Step S310, calculating the second distortion loss according to the non-critical multi-view image and the non-critical scene representation model;

[0141] Exemplarily, the second distortion loss D is calculated by the following formula: (t) :

[0142] D (t) =(1-λ ssim )×L1+λ ssim ×(1-SSIM)

[0143] Among them, λ ssim The weight used to weigh the L1 loss and SSIM loss, where L1 is the absolute error and SSIM is a parameter determined based on the similarity between the non-critical multi-view images and the non-critical scene representation model.

[0144] Step S320: Calculate the second bit rate loss according to the number and frequency of binary numbers in the preset motion field and the preset characteristic residual field.

[0145] Exemplarily, the second code rate loss R is calculated by the following formula: (t) :

[0146] R (t) =M + ×(-log2(h f ))+M - ×(-log2(1-h f ))

[0147] Among them, M + and M - They represent the binary multi-level hash grids storing the motion field and the feature residual field, that is, the number of “+1” and “-1” in the preset hash table is the number of existence, h f It indicates the frequency of occurrence of the "+1" symbol, that is, the existence frequency.

[0148] In this implementation, by calculating the second distortion loss and the second code rate loss, an effective data basis is provided for comprehensively considering the reconstruction quality and the data transmission cost.

[0149] Step S302, optimizing the preset motion field, the preset feature residual field and the MLP according to the second joint loss to obtain an optimized motion field, an optimized feature residual field and a trained MLP;

[0150] It is understandable that since the preset motion field, feature residual field and MLP often cannot adapt to the reconstruction requirements of non-critical scenes very well, performing step S302 can avoid reconstruction errors caused by mismatch between the motion field, feature residual field and MLP, thereby improving the accuracy and robustness of non-critical scene reconstruction.

[0151] Exemplarily, the calculated joint loss is minimized by continuously adjusting the parameters of the motion field and feature residual field cached in the binary multi-level hash grid, as well as the weights and biases of each layer of the neural network in the MLP, so that the motion field, feature residual field and MLP that minimize the joint loss are used as the optimized motion field and feature residual field, as well as the trained MLP.

[0152] Step S303, based on the optimized motion field and the trained MLP, returning to the step of interpolating the decoded motion field according to the coordinates of each anchor point;

[0153] Exemplarily, three-dimensional linear interpolation is performed on the optimized motion field according to the coordinates of each anchor point to obtain each motion characterization feature vector, and each motion characterization feature vector is input into the trained MLP so that the MLP can more accurately predict the offset change between the position offset of each Gaussian point at the current moment and the position offset of each Gaussian point in the scene representation model at the previous moment.

[0154] Step S304, based on the optimized feature residual field and the trained MLP, returning to the step of interpolating the decoded feature residual field according to the coordinates of each anchor point.

[0155] Exemplarily, three-dimensional linear interpolation is performed on the optimized feature residual field according to the coordinates of each anchor point to obtain a feature vector representing each feature residual, and each feature residual characterization feature vector is input into the trained MLP so that the MLP can more accurately predict the feature residuals between each current feature vector and each feature vector in the scene representation model at the previous moment.

[0156] In this embodiment, by performing joint loss optimization on the motion field, characteristic residual field and MLP, distortion problems, low coding efficiency and waste of computing resources in the reconstruction of non-critical scenes are avoided, and the coding efficiency is improved and the computational complexity is reduced while ensuring the reconstruction quality, thereby optimizing the reconstruction process of non-critical scenes to enhance the performance of the overall three-dimensional volumetric video encoding and generation system.

[0157] For example, to help understand the implementation process of the 3D volume video encoding and generation method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 7 , Figure 7A brief flowchart of a three-dimensional volume video encoding and generation method is provided, specifically:

[0158] The multi-view image corresponding to time t=0 (starting time) is the key multi-view image, and the viewpoint images corresponding to subsequent times such as t=1, t=2, etc. are non-key multi-view images. For the key multi-view image, the key scene is first reconstructed in the encoder to obtain the key scene represented by a three-dimensional Gaussian, and the key scene is entropy encoded to obtain the binary code stream of the key scene. The binary code stream of the key scene is decoded in the decoder to obtain the key scene representation model. For the non-key multi-view image at any subsequent time, the preset transformation buffer is encoded to obtain the binary code stream of the non-key scene. The binary code stream of the non-key scene is decoded in the decoder, and the non-key scene representation model is obtained by combining the scene representation model of the previous time. Finally, each scene representation model is rendered to obtain a three-dimensional volume video for user interaction, such as adjustment of the camera posture and control of the playback time.

[0159] For further information, please refer to Figure 8 , Figure 8 A schematic diagram of a coding framework of a three-dimensional volume video coding and generation method is provided, specifically:

[0160] For the key multi-view image at t=0, the camera is calibrated to obtain a sparse point cloud, and the anchor-based Gaussian representation model is initialized based on the sparse point cloud to obtain the key scene. The key scene is quantized Q, entropy encoded, and dequantized DQ through the context model to obtain the decoded key scene representation model S0. For the non-key multi-view image at t=1, in the binary transform cache, based on the transformation anchor point parameters predicted from the hash table, combined with the key scene representation model, the non-key scene representation model S1 is obtained by changing the anchor point parameter operation T, and the operation at time t=1 is repeated for subsequent times t=i. The present application adopts an anchor-based Gaussian representation method, that is, the scene representation model S is processed by MLP i The anchor point attributes are: anchor point coordinate x, feature vector f, size f and Gaussian point position offset o, and the output is based on the three-dimensional Gaussian: opacity α, three primary color information c, scale s, quaternion r, and view A represented by mean μ.

[0161] This application has carried out comparative analysis on the N3DV dataset with the existing volumetric video online reconstruction methods StreamRF and 3DGStream. The objective results are shown in Table 1 below, where PSNR represents peak signal-to-noise ratio, Storage represents the storage space occupied, Train represents the training time, and Render represents the video frame rate after rendering. The experimental results show that the present invention achieves the highest rendering quality at the lowest storage cost, and the training time is similar to that of the StreamRF and 3DGStream methods, supporting real-time rendering (FPS>30). In the above experiment, a one-stage training method was adopted, and a transform cache model was used to simultaneously characterize the Gaussian point position offset and feature residual, and the binary hash grid was implemented based on three planes.

[0162] method PSNR(dB) Storage(MB) Train(mins) Render(FPS) StreamRF 30.68 31.4 0.23 8.3 3DGStream 31.67 7.8 0.20 215 This application 32.32 0.09 0.22 126

[0163] Table 1

[0164] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the three-dimensional volumetric video encoding and generation method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0165] This application also provides a three-dimensional volume video encoding and generation device, please refer to Fig. 9 , the three-dimensional volume video encoding and generating device comprises:

[0166] An acquisition module 10 is used to acquire a multi-view video captured by multiple cameras, and determine a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video;

[0167] A first encoding module 20 is used to reconstruct the key scene of the multi-view video according to the key multi-view image, and encode the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video;

[0168] The second encoding module 30 is used to encode the preset transformation buffer for any non-critical multi-view image to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain a non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model of the previous moment;

[0169] The generating module 40 is configured to generate a three-dimensional volumetric video based on the key scene representation model and each non-key scene representation model after traversing each non-key multi-viewpoint image.

[0170] Optionally, the first encoding module 20 is further used for:

[0171] Obtaining anchor point attributes of each anchor point in the key scene;

[0172] The attributes of each anchor point are quantized by using a preset scene compression model, and entropy coding is performed on the quantized attributes of each anchor point to obtain a code stream of the key scene.

[0173] Optionally, the first optimization module 50 in the three-dimensional volume video encoding and generating device is used to:

[0174] Calculating a first distortion loss and a first bit rate loss in the key scene reconstruction process, and constructing a first joint loss based on the first distortion loss and the first bit rate loss;

[0175] The anchor point attributes and the scene compression model are optimized according to the first joint loss to obtain adjusted anchor point attributes and a trained scene compression model, and based on the adjusted anchor point attributes and the trained scene compression model, the step of quantizing each anchor point attribute through the preset scene compression model is performed.

[0176] Optionally, the first optimization module 50 is further configured to:

[0177] Calculating the first distortion loss according to the key multi-viewpoint image and the key scene representation model;

[0178] The first bit rate loss is calculated according to the probability distribution of the anchor point attribute, wherein the probability distribution is predicted by the preset scene compression model.

[0179] Optionally, the preset transformation cache includes a preset motion field, a preset feature residual field and a preset multi-layer perceptron MLP, and the second encoding module 30 is further used for:

[0180] Encoding the preset motion field, the preset feature residual field and the preset multi-layer perceptron MLP to obtain a code stream of the non-critical scene;

[0181] When decoding the bitstream of the non-critical scene, obtaining the anchor point coordinates of each anchor point in the scene corresponding to the multi-viewpoint image at the previous moment;

[0182] Interpolate the decoded motion field according to the coordinates of each anchor point to obtain each motion representation feature vector, and input the motion representation feature vector into the decoded MLP to predict the offset change between the position offset of each Gaussian point at the current moment and the position offset of each Gaussian point in the scene representation model at the previous moment through the MLP;

[0183] Interpolate the decoded feature residual field according to the coordinates of each anchor point to obtain a feature vector representing each feature residual, and input the feature vector representing each feature residual into the decoded MLP to predict the feature residual between each current feature vector and each feature vector in the scene representation model at the previous moment through the MLP;

[0184] Each offset change is combined with the corresponding Gaussian point position offset in the scene representation model at the previous moment, and each feature residual is combined with the corresponding feature vector in the scene representation model at the previous moment to obtain the non-critical scene representation model.

[0185] Optionally, the second optimization module 60 in the three-dimensional volume video encoding and generating device is used to:

[0186] Calculating a second distortion loss and a second bit rate loss in the reconstruction process of the non-critical scene, and constructing a second joint loss based on the second distortion loss and the second bit rate loss;

[0187] Optimizing the preset motion field, the preset feature residual field, and the MLP according to the second joint loss to obtain an optimized motion field, an optimized feature residual field, and a trained MLP;

[0188] Based on the optimized motion field and the trained MLP, returning to the step of interpolating the decoded motion field according to the coordinates of each anchor point;

[0189] Based on the optimized feature residual field and the trained MLP, return to the step of interpolating the decoded feature residual field according to the coordinates of each anchor point.

[0190] Optionally, the second optimization module 60 is further configured to:

[0191] Calculating the second distortion loss according to the non-critical multi-viewpoint image and the non-critical scene representation model;

[0192] The second bit rate loss is calculated according to the number and frequency of binary numbers in the preset motion field and the preset characteristic residual field.

[0193] The 3D volumetric video encoding and generation device provided by the present application adopts the 3D volumetric video encoding and generation method in the above-mentioned embodiment, which can solve the technical problem of how to efficiently encode and generate 3D volumetric video. Compared with the prior art, the beneficial effects of the 3D volumetric video encoding and generation device provided by the present application are the same as the beneficial effects of the 3D volumetric video encoding and generation method provided by the above-mentioned embodiment, and other technical features in the 3D volumetric video encoding and generation device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0194] The present application provides an electronic device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the three-dimensional volume video encoding and generation method in the above-mentioned embodiment 1.

[0195] Reference below Fig.10 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0196] like Fig.10As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0197] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0198] The electronic device provided by the present application adopts the three-dimensional volumetric video encoding and generation method in the above embodiment, which can solve the technical problem of how to efficiently encode and generate three-dimensional volumetric video. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the three-dimensional volumetric video encoding and generation method provided by the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0199] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0200] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the three-dimensional volume video encoding and generation method in the above-mentioned embodiment.

[0201] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0202] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: obtains a multi-view video shot by multiple cameras, and determines a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video; reconstructs a key scene of the multi-view video according to the key multi-view image, and encodes the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video; for any non-key multi-view image, encodes a preset transformation cache to obtain a code stream of the non-key scene corresponding to the non-key multi-view image, so that after decoding, it is combined with the scene representation model of the previous moment to obtain a non-key scene representation model of the multi-view video; after traversing each non-key multi-view image, generates a three-dimensional volumetric video based on the key scene representation model and each non-key scene representation model.

[0203] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0204] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0205] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned three-dimensional volumetric video encoding and generation method, and can solve the technical problem of how to efficiently encode and generate three-dimensional volumetric video. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the three-dimensional volumetric video encoding and generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0206] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned three-dimensional volume video encoding and generation method when executed by a processor.

[0207] The computer program product provided by the present application can solve the technical problem of how to efficiently encode and generate a three-dimensional volumetric video. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the three-dimensional volumetric video encoding and generation method provided by the above embodiment, and will not be repeated here.

Claims

1. A three-dimensional volume video encoding and generation method, characterized in that: The three-dimensional volume video encoding and generation method comprises: Acquire a multi-view video captured by multiple cameras, and determine a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video; Reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video; For any non-critical multi-view image, encode the preset transformation buffer to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain a non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model of the previous moment; After traversing each non-critical multi-viewpoint image, a three-dimensional volume video is generated based on the critical scene representation model and each non-critical scene representation model.

2. The three-dimensional volumetric video encoding and generation method according to claim 1, characterized in that: The step of encoding the key scene to obtain a code stream of the key scene includes: Obtaining anchor point attributes of each anchor point in the key scene; The attributes of each anchor point are quantized by using a preset scene compression model, and entropy coding is performed on the quantized attributes of each anchor point to obtain a code stream of the key scene.

3. The three-dimensional volumetric video encoding and generation method according to claim 2, characterized in that: The three-dimensional volume video encoding and generation method also includes: Calculating a first distortion loss and a first bit rate loss in the key scene reconstruction process, and constructing a first joint loss based on the first distortion loss and the first bit rate loss; The anchor point attributes and the scene compression model are optimized according to the first joint loss to obtain adjusted anchor point attributes and a trained scene compression model, and based on the adjusted anchor point attributes and the trained scene compression model, the step of quantizing each anchor point attribute through the preset scene compression model is performed.

4. The three-dimensional volumetric video encoding and generation method according to claim 3, characterized in that: The step of calculating the first distortion loss and the first bit rate loss in the key scene reconstruction process comprises: Calculating the first distortion loss according to the key multi-viewpoint image and the key scene representation model; The first bit rate loss is calculated according to the probability distribution of the anchor point attribute, wherein the probability distribution is predicted by the preset scene compression model.

5. The three-dimensional volume video encoding and generation method according to claim 1, characterized in that: The preset transformation cache includes a preset motion field, a preset feature residual field and a preset multi-layer perceptron MLP. The step of encoding the preset transformation cache to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image for decoding and combining the scene representation model at the previous moment to obtain the non-critical scene representation model of the multi-view video includes: Encoding the preset motion field, the preset feature residual field and the preset multi-layer perceptron MLP to obtain a code stream of the non-critical scene; When decoding the bitstream of the non-critical scene, obtaining the anchor point coordinates of each anchor point in the scene corresponding to the multi-viewpoint image at the previous moment; Interpolate the decoded motion field according to the coordinates of each anchor point to obtain each motion representation feature vector, and input the motion representation feature vector into the decoded MLP to predict the offset change between the position offset of each Gaussian point at the current moment and the position offset of each Gaussian point in the scene representation model at the previous moment through the MLP; Interpolate the decoded feature residual field according to the coordinates of each anchor point to obtain a feature vector representing each feature residual, and input the feature vector representing each feature residual into the decoded MLP to predict the feature residual between each current feature vector and each feature vector in the scene representation model at the previous moment through the MLP; Each offset change is combined with the corresponding Gaussian point position offset in the scene representation model at the previous moment, and each feature residual is combined with the corresponding feature vector in the scene representation model at the previous moment to obtain the non-critical scene representation model.

6. The three-dimensional volume video encoding and generation method according to claim 5, characterized in that: The three-dimensional volume video encoding and generation method also includes: Calculating a second distortion loss and a second bit rate loss in the reconstruction process of the non-critical scene, and constructing a second joint loss based on the second distortion loss and the second bit rate loss; Optimizing the preset motion field, the preset feature residual field, and the MLP according to the second joint loss to obtain an optimized motion field, an optimized feature residual field, and a trained MLP; Based on the optimized motion field and the trained MLP, returning to the step of interpolating the decoded motion field according to the coordinates of each anchor point; Based on the optimized feature residual field and the trained MLP, return to the step of interpolating the decoded feature residual field according to the coordinates of each anchor point.

7. The three-dimensional volumetric video encoding and generation method according to claim 6, characterized in that: The step of calculating the second distortion loss and the second bit rate loss in the reconstruction process of the non-critical scene comprises: Calculating the second distortion loss according to the non-critical multi-viewpoint image and the non-critical scene representation model; The second bit rate loss is calculated according to the number and frequency of binary numbers in the preset motion field and the preset characteristic residual field.

8. A three-dimensional volume video encoding and generation device, characterized in that: The three-dimensional volume video encoding and generating device comprises: An acquisition module, used to acquire a multi-view video captured by multiple cameras, and determine a key multi-view image at a starting moment and non-key multi-view images at subsequent moments in the multi-view video; A first encoding module is used to reconstruct a key scene of the multi-view video according to the key multi-view image, and encode the key scene to obtain a code stream of the key scene for decoding to obtain a key scene representation model of the multi-view video; The second key module is used to encode the preset transformation buffer for any non-critical multi-view image to obtain a code stream of a non-critical scene corresponding to the non-critical multi-view image, so as to obtain a non-critical scene representation model of the multi-view video after decoding and combining it with the scene representation model of the previous moment; A generation module is used to generate a three-dimensional volume video based on the key scene representation model and each non-key scene representation model after traversing each non-key multi-viewpoint image.

9. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the three-dimensional volume video encoding and generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the three-dimensional volume video encoding and generation method according to any one of claims 1 to 7 are implemented.

Images