Three-dimensional volumetric video encoding and generation method, apparatus, device, and medium

Abstract

The application discloses a three-dimensional volume video coding and generation method and device, electronic equipment and a storage medium, and relates to the technical field of volume video, and comprises the following steps: acquiring a key multi-view image in multi-view video; reconstructing a key scene of the multi-view video according to the key multi-view image, and coding the key scene to obtain a code stream of the key scene, so as to obtain a key scene representation model by decoding; coding a preset transformation cache to obtain a code stream of a non-key scene, so as to obtain a non-key scene representation model by combining the scene representation model of the last moment after decoding; and generating a three-dimensional volume video based on the key scene representation model and the non-key scene representation models after traversing all non-key multi-view images. The application realizes the effect of quickly, accurately and high-quality generation of the three-dimensional volume video, significantly improves the data processing efficiency of the volume video generation process, and reduces the data transmission amount.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of volumetric video, and particularly to a three-dimensional volumetric video encoding and generating method and device, electronic equipment and storage medium. BACKGROUND

[0002] Three-dimensional volumetric video technology aims to reconstruct the structure of a dynamic scene to accurately present the dynamic changes and spatial layout of the scene, and is widely used in virtual reality, augmented reality and other fields to provide realistic immersive experience for users. Nowadays, with the rapid development of virtual reality and augmented reality applications, we need to quickly, accurately and high-quality reconstruct the three-dimensional volumetric video of the scene from the input multi-view two-dimensional images or videos, so as to provide good visual experience. However, in the existing volumetric video generation methods, the modeling and compression of scenes at different times usually cause high computational complexity and slow scene reconstruction speed.

[0003] Therefore, how to efficiently encode and generate three-dimensional volumetric video is a problem to be solved at present. SUMMARY

[0004] The main purpose of the present application is to provide a three-dimensional volumetric video encoding and generating method and device, electronic equipment and storage medium, aiming to solve the technical problem of how to efficiently encode and generate three-dimensional volumetric video.

[0005] To achieve the above purpose, the present application provides a three-dimensional volumetric video encoding and generating method, which comprises:

[0006] Obtaining a multi-view video shot by multiple cameras, and determining a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video;

[0007] Reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene, so as to decode to obtain a key scene representation model of the multi-view video;

[0008] For any non-key multi-view image, encoding a preset transformation cache to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to combine the scene representation model at the previous time after decoding to obtain a non-key scene representation model of the multi-view video;

[0009] After traversing all non-key multi-view images, generating a three-dimensional volumetric video based on the key scene representation model and the non-key scene representation models.

[0010] In an embodiment, the step of encoding the key scene to obtain a bitstream of the key scene comprises:

[0011] obtaining anchor point attributes of each anchor point in the key scene;

[0012] quantizing each anchor point attribute by using a preset scene compression model, and performing entropy encoding on the quantized anchor point attributes to obtain the bitstream of the key scene.

[0013] In an embodiment, the three-dimensional volumetric video encoding and generation method further comprises:

[0014] calculating a first distortion loss and a first code rate loss in the key scene reconstruction process, and constructing a first joint loss based on the first distortion loss and the first code rate loss;

[0015] optimizing the anchor point attributes and the scene compression model according to the first joint loss to obtain adjusted anchor point attributes and a trained scene compression model, and performing the step of quantizing each anchor point attribute by using a preset scene compression model based on the adjusted anchor point attributes and the trained scene compression model.

[0016] In an embodiment, the step of calculating the first distortion loss and the first code rate loss in the key scene reconstruction process comprises:

[0017] calculating the first distortion loss according to the key multi-view image and the key scene representation model;

[0018] calculating the first code rate loss according to a probability distribution of the anchor point attributes, wherein the probability distribution is predicted by the preset scene compression model.

[0019] In an embodiment, the preset transformation cache comprises a preset motion field, a preset feature residual field, and a preset multi-layer perception (MLP), and the step of encoding the preset transformation cache to obtain a bitstream of a non-key scene corresponding to the non-key multi-view image, so as to obtain a non-key scene representation model of the multi-view video by combining a scene representation model of the last time after decoding comprises:

[0020] encoding the preset motion field, the preset feature residual field, and the preset multi-layer perception (MLP) to obtain the bitstream of the non-key scene;

[0021] when decoding the bitstream of the non-key scene, obtaining anchor point coordinates of each anchor point in a scene corresponding to a multi-view image of the last time;

[0022] interpolating the decoded motion field according to the anchor point coordinates to obtain motion representation feature vectors, and inputting the motion representation feature vectors into the decoded MLP to predict motion field changes between the current time instant and the previous time instant of the scene representation model through the MLP;

[0023] interpolating the decoded feature residual field according to the anchor point coordinates to obtain feature residual representation feature vectors, and inputting the feature residual representation feature vectors into the decoded MLP to predict feature residuals between the current feature vectors and the previous time instant of the scene representation model through the MLP;

[0024] combining the motion field changes and the corresponding Gaussian point position offsets in the previous time instant of the scene representation model, and combining the feature residuals and the corresponding feature vectors in the previous time instant of the scene representation model to obtain the non-key scene representation model.

[0025] In an embodiment, the three-dimensional volumetric video encoding and generation method further comprises:

[0026] calculating a second distortion loss and a second code rate loss in the non-key scene reconstruction process, and constructing a second joint loss based on the second distortion loss and the second code rate loss;

[0027] optimizing the preset motion field, the preset feature residual field, and the MLP according to the second joint loss to obtain an optimized motion field, an optimized feature residual field, and a trained MLP;

[0028] based on the optimized motion field and the trained MLP, returning to perform the step of interpolating the decoded motion field according to the anchor point coordinates;

[0029] based on the optimized feature residual field and the trained MLP, returning to perform the step of interpolating the decoded feature residual field according to the anchor point coordinates.

[0030] In an embodiment, the step of calculating the second distortion loss and the second code rate loss in the non-key scene reconstruction process comprises:

[0031] calculating the second distortion loss according to the non-key multi-view image and the non-key scene representation model;

[0032] calculating the second code rate loss according to the number and frequency of existing binary numbers in the preset motion field and the preset feature residual field.

[0033] In addition, to achieve the above object, the application further provides a three-dimensional volumetric video encoding and generating device, which comprises:

[0034] an acquisition module, configured to acquire a multi-view video captured by a multi-camera, and determine a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video;

[0035] a first encoding module, configured to reconstruct a key scene of the multi-view video according to the key multi-view image, and encode the key scene to obtain a code stream of the key scene, so as to decode to obtain a key scene representation model of the multi-view video;

[0036] a second key module, configured to, for any non-key multi-view image, encode a preset transform cache to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to, after decoding, combine a scene representation model at a previous time to obtain a non-key scene representation model of the multi-view video;

[0037] a generating module, configured to, after traversing all the non-key multi-view images, generate a three-dimensional volumetric video based on the key scene representation model and the non-key scene representation models.

[0038] In addition, to achieve the above object, the application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the three-dimensional volumetric video encoding and generating method as described above.

[0039] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the three-dimensional volumetric video encoding and generating method as described above.

[0040] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the three-dimensional volumetric video encoding and generating method as described above.

[0041] The one or more technical solutions provided by the application have at least the following technical effects:

[0042] The application first acquires a multi-view video shot by multiple cameras, and determines a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video, so as to effectively identify and separate out the key multi-view images which are crucial to scene reconstruction and the non-key multi-view images which are used to describe dynamic changes, and provide accurate time sequence data basis for subsequent scene reconstruction; the key scene of the multi-view video is reconstructed according to the key multi-view images, and the key scene is encoded to obtain a code stream of the key scene, so as to obtain a key scene representation model of the multi-view video by decoding, a three-dimensional scene model is constructed by processing the key multi-view images, and the three-dimensional scene model is encoded to generate a compressed three-dimensional model code stream which can accurately represent the key scene, so that the original key scene can be quickly and high-quality recovered during decoding, and a stable reconstruction starting point is provided for the entire video sequence; for any non-key multi-view image, a preset transformation buffer is encoded to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to obtain a non-key scene representation model of the multi-view video by combining the scene representation model at the previous time after decoding, by only encoding the transformation buffer, the dynamic change part between frames is efficiently captured and encoded, so that a three-dimensional representation of the non-key scene can be quickly generated by a small amount of parameter adjustment during decoding, and the reconstruction speed is significantly improved and the transmission data amount is reduced; after traversing each non-key multi-view image, a three-dimensional volume video is generated based on the key scene representation model and each non-key scene representation model, the key scene representation model and the non-key scene representation model are combined by three-dimensional video synthesis technology to generate a complete three-dimensional volume video, the video can realistically reproduce the dynamic changes and spatial layout of the original scene, and provides a high-quality immersive experience for users.

[0043] In conclusion, in the reconstruction process of the non-key scene, the preset transformation buffer is combined with the scene representation model at the previous time, so that the non-key scene is quickly reconstructed, the problems of high computational complexity and slow scene reconstruction speed caused by complete modeling and compression of the scene at each time in the traditional volume video generation method are avoided, the effect of quickly, accurately and high-quality generating a three-dimensional volume video is realized, a realistic immersive experience is provided for users, the data processing efficiency of the volume video generation process is significantly improved, and the data transmission amount after encoding is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those drawings can also help the ordinary skilled in the art to obtain other drawings without any creative effort.

[0046] Figure 1 A flowchart provided for the three-dimensional volume video coding and generating method of the first embodiment of the present application;

[0047] Figure 2 A first buffer diagram of the three-dimensional volume video coding and generating method provided for the first embodiment of the present application;

[0048] Figure 3 A second buffer diagram of the three-dimensional volume video coding and generating method provided for the first embodiment of the present application;

[0049] Figure 4 A flowchart provided for the three-dimensional volume video coding and generating method of the second embodiment of the present application;

[0050] Figure 5 A hash grid diagram of the three-dimensional volume video coding and generating method provided for the second embodiment of the present application;

[0051] Figure 6 A flowchart provided for the three-dimensional volume video coding and generating method of the third embodiment of the present application;

[0052] Figure 7 A brief flowchart of the three-dimensional volume video coding and generating method provided for the third embodiment of the present application;

[0053] Figure 8 An encoding framework diagram of the three-dimensional volume video coding and generating method provided for the third embodiment of the present application;

[0054] Figure 9 A module structure diagram of the three-dimensional volume video coding and generating device of the present application;

[0055] Figure 10 A device structure diagram of the hardware running environment related to the three-dimensional volume video coding and generating method in the embodiments of the present application.

[0056] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and not to limit the present application.

[0058] For better understanding of the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0059] The main solution of the embodiment of the present application is: acquiring a multi-view video captured by multiple cameras, and determining a key multi-view image at a starting time and non-key multi-view images at subsequent multiple times in the multi-view video; reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene, so as to decode to obtain a key scene representation model of the multi-view video; for any non-key multi-view image, predicting a non-key scene corresponding to the non-key multi-view image through a preset artificial neural network model, a transformation anchor point parameter between the non-key scene and a scene corresponding to a multi-view image at a previous time, and encoding the transformation anchor point parameter to obtain a code stream of the non-key scene, so as to obtain a non-key scene representation model of the multi-view video after decoding in combination with a scene representation model at the previous time; and generating a three-dimensional volume video based on the key scene representation model and the non-key scene representation models after traversing each non-key multi-view image.

[0060] Due to the rapid development of virtual reality and augmented reality applications, we need to quickly, accurately and high-quality reconstruct a three-dimensional volume video of a scene from input multi-view two-dimensional images or videos, so as to provide a good visual experience. However, in the existing volume video generation method, because the scenes at different times need to be modeled and compressed, the calculation complexity is high and the scene reconstruction speed is slow. Therefore, how to efficiently encode and generate a three-dimensional volume video is a problem that needs to be solved at present.

[0061] The present application provides a solution, which realizes the fast reconstruction of a non-key scene by combining a preset transformation cache with a scene representation model at a previous time in the reconstruction process of the non-key scene, thereby avoiding the problems of high calculation complexity and slow scene reconstruction speed caused by complete modeling and compression of the scene at each time in the traditional volume video generation method, realizing the effect of quickly, accurately and high-quality generating a three-dimensional volume video, providing a realistic immersive experience for users, significantly improving the data processing efficiency of the volume video generation process, and reducing the data transmission amount after encoding.

[0062] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions. The embodiments and the following embodiments will be described below taking the electronic device as an example.

[0063] Based on this, the embodiment of the present application provides a three-dimensional volume video coding and generation method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the three-dimensional volume video coding and generation method of the present application is shown in the figure.

[0064] In this embodiment, the three-dimensional volume video coding and generation method comprises steps S10-S40:

[0065] Step S10, acquiring a multi-view video shot by multiple cameras, and determining a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video;

[0066] It should be noted that the multi-view video refers to a video shot by multiple cameras from multiple different angles at the same time, which can be used to reconstruct the three-dimensional information of the scene; the key multi-view image refers to a key frame in the multi-view video used to reconstruct the initial state of the three-dimensional scene, which contains rich scene information and details; the non-key multi-view image refers to other frames except the key frame, which are used to describe the dynamic changes of the scene over time.

[0067] It can be understood that, since the traditional volume video generation method needs to model and compress the scenes at different times, i.e., the same processing is performed for different multi-view images, step S10 is performed to avoid the problem of complex calculation caused by taking the multi-view images at different times as the data basis for modeling, by distinguishing the key image at the starting time and the non-key images at the subsequent times, providing a reliable data basis for different processing of different images, and further reducing the complex processing operation of the non-key images.

[0068] Illustratively, a multi-view video shot by multiple cameras is acquired, and an image at t=0 in the video is determined as a key multi-view image, and images at t>0 (such as t=1, t=2, …) in the video are determined as non-key multi-view images.

[0069] Step S20, reconstructing a key scene of the multi-view video according to the key multi-view image, and encoding the key scene to obtain a code stream of the key scene, for decoding to obtain a key scene representation model of the multi-view video;

[0070] It should be noted that the key scene refers to a static model of the three-dimensional scene reconstructed by the key multi-view image, which represents a key state in the video sequence; the code stream of the key scene refers to the compressed data stream obtained after encoding the key scene, which is used for transmission and storage; the key scene representation model refers to the model decoded from the code stream of the key scene, which can represent the three-dimensional structure and appearance of the scene.

[0071] It can be understood that, since at least an accurate initial scene model needs to be established for the reconstruction of the scene, step S20 is performed to avoid the inaccuracy of subsequent scene reconstruction, thereby providing an effective reconstruction platform for realizing high-quality three-dimensional scene reconstruction.

[0072] Exemplarily, camera calibration is performed on the key multi-view images by the Colmap method to obtain sparse point clouds and internal and external parameters of each camera, and a Gaussian representation model based on anchor points is initialized by using the coefficient point clouds, and then the properties of the anchor points are continuously optimized according to the key multi-view images, and the internal and external parameters obtained are combined to realize the best key scene reconstruction.

[0073] In a feasible implementation, the step of encoding the key scene to obtain the code stream of the key scene in step S20 can include steps S21-S22:

[0074] In step S21, the properties of the anchor points in the key scene are obtained.

[0075] It should be noted that the anchor points refer to reference points representing specific positions or points in a three-dimensional scene, which are key points in scene reconstruction and are used to describe the structure of the scene; the properties of the anchor points refer to state information related to the anchor points, including anchor point coordinates, feature vectors, sizes, and Gaussian point position offsets, which help to define the specific state of the anchor points in the three-dimensional space.

[0076] In step S22, the properties of the anchor points are quantized by a preset scene compression model, and the quantized properties of the anchor points are entropy encoded to obtain the code stream of the key scene.

[0077] It should be noted that the preset scene compression model is a way of compressing and encoding the reconstructed scene or compressing the scene while reconstructing the scene, which can be a context model, a hyper-prior model, etc., wherein the context model can include a hash grid and an MLP (Multilayer Perceptron), which is used to predict the quantization step and probability distribution of each anchor point property according to the context information (such as the properties of adjacent anchor points) stored in the hash grid. The predicted quantization step is used to quantize the properties of each anchor point, and the predicted probability distribution is used to calculate the code rate loss in the scene reconstruction process. The optimization of the quantization process of the anchor point properties according to the context information of the scene can avoid the quantization error caused by not considering the context information, realize more accurate representation of the anchor point properties, and improve the accuracy and efficiency of the quantization process, thereby improving the three-dimensional scene reconstruction.

[0078] It can be understood that, since the scene needs to be compressed and quantized in the process of transmission, and the quantized anchor attribute data still has redundancy, step S22 is performed to quantize and entropy encode the anchor attribute, so as to effectively compress the anchor attribute data, reduce the amount of data required for storage and transmission, avoid waste of bandwidth and storage resources caused by transmission of uncompressed data, realize efficient data compression, thereby significantly reducing the data size and improving the efficiency of data transmission and storage while maintaining the necessary information of the reconstructed scene.

[0079] For example, the quantization step of each anchor attribute is predicted according to the context model, the anchor attribute is quantized according to the quantization step output by the context model, and then the quantized anchor attribute is entropy encoded by using an arithmetic encoder to obtain the binary code stream corresponding to the key scene.

[0080] In this embodiment, by quantizing and entropy encoding the anchor attribute, the data redundancy problem in the data transmission process is avoided, efficient compression and accurate representation of the anchor attribute of the key scene are realized, thereby improving the efficiency of data transmission and storage while ensuring the quality of three-dimensional scene reconstruction.

[0081] In step S30, for any non-key multi-view image, the preset transform buffer is encoded to obtain the code stream of the non-key scene corresponding to the non-key multi-view image, so as to obtain the non-key scene representation model of the multi-view video by combining the scene representation model of the previous time after decoding.

[0082] It should be noted that the preset transform buffer is a data structure which at least stores binary transform parameters for encoding the non-key multi-view image, including the motion field and the feature residual field, and a pre-configured multi-layer perception (MLP), which can be used to describe the changes between adjacent frame images and the dynamic characteristics of the scene. For different three-dimensional dynamic scenes, the transform buffer has different hash grid structures. Please refer to Figure 4 The 3D grid structure A can be used alone, the 2D grid structure B representing three planes can be used, or the 2D-3D grid composite structure C can be used. In this embodiment, the hash grid structure of the transform buffer is not specifically limited; the code stream of the non-key scene refers to the compressed data stream obtained by encoding the transform buffer of the non-key scene; and the non-key scene representation model refers to the model obtained by decoding the scene representation model of the previous time and the code stream of the current non-key scene, which is used to represent the scene state at the non-key time.

[0083] In addition, it should be noted that the scene representation model of the previous time can be a key scene representation model or a non-key scene representation model.

[0084] It can be understood that, due to the small change of the non-key multi-view image relative to the key frame, in order to efficiently describe the dynamic change of the scene, step S30 is performed, the calculation burden of the complete reconstruction of each non-key frame, i.e. the non-key multi-view image, can be avoided, by only encoding the preset transformation cache, to obtain the scene representation model at the current time by combining the scene representation model at the last time after decoding, the time of encoding and decoding and the required storage space are reduced, thereby realizing fast and accurate scene dynamic representation.

[0085] Exemplarily, for any non-key multi-view image at time t, the preset motion field, the preset feature residual field and the preset MLP in the preset transformation cache are encoded to obtain the code stream of the non-key scene, wherein the preset motion field and the preset feature residual field are stored in the transformation cache in the form of a binary hash table, which replaces the feature type from a 32-bit floating point number to a 1-bit binary number, greatly reducing the storage occupation of the transformation anchor point parameter cache. After decoding the code stream, the anchor point coordinates of the scene representation model at the last time are used to interpolate the decoded motion field and feature residual field, and the interpolation result is predicted by the decoded MLP to obtain the partial anchor point attribute change between the scene representation model at the current time and the scene representation model at the last time, including the offset change of the position of each Gaussian point and the feature residual, so as to obtain the scene representation model at the current time according to the anchor point attribute of the scene representation model at the last time and the partial anchor point attribute change.

[0086] In a feasible implementation, the preset transformation cache includes a preset motion field, a preset feature residual field and a preset multi-layer perception MLP, and the step of encoding the preset transformation cache in step S30 to obtain the code stream of the non-key scene corresponding to the non-key multi-view image, for combining the scene representation model at the last time after decoding to obtain the non-key scene representation model of the multi-view video, can include steps S31-S35:

[0087] Step S31, encoding the preset motion field, the preset feature residual field and the preset multi-layer perception MLP to obtain the code stream of the non-key scene;

[0088] It should be noted that the preset motion field refers to a predefined field, which contains the motion mode that can appear in the scene, and is used to predict the position offset of the Gaussian point; the preset feature residual field refers to a predefined field, which contains the change mode of the feature vector in the scene, and is used to predict the residual of the feature vector.

[0089] Step S32, when decoding the code stream of the non-key scene, obtaining the anchor point coordinates of each anchor point in the scene corresponding to the multi-view image at the last time;

[0090] It should be noted that the scene corresponding to the multi-view image at the previous moment can be a key scene or a non-key scene; the anchor point coordinates refer to the specific position coordinates of the anchor point in the three-dimensional space, which is the basic reference point for scene reconstruction.

[0091] Step S33, according to the anchor point coordinates, the decoded motion field is interpolated to obtain each motion representation feature vector, and the motion representation feature vector is input into the decoded MLP to predict the displacement change between the position offset of each Gaussian point at the current moment and the position offset of each Gaussian point in the scene representation model at the previous moment through the MLP.

[0092] It should be noted that the motion representation feature vector refers to a vector used to describe the motion characteristics of the Gaussian point, which contains the information of the position offset of the Gaussian point.

[0093] It can be understood that since the motion of the Gaussian point in the scene needs to be predicted, step S33 is performed to avoid directly calculating the specific offset of all Gaussian points, thereby reducing the amount of calculation, realizing efficient prediction of the Gaussian point offset in the scene, and improving the efficiency and accuracy of dynamic scene reconstruction.

[0094] Exemplarily, according to the anchor point coordinates x, the preset motion field M t in the preset transformation cache is interpolated to obtain the motion representation feature vector interp(x, M t ), and the motion representation feature vector is input into the MLP to predict the displacement change Δo t of the position offset of the Gaussian point:

[0095] Δo t =MLP1(interp(x, M t ))

[0096] Wherein, the displacement change Δo t is used to combine the corresponding position offset o t-1 of the Gaussian point in the scene representation model at the previous moment to obtain the position offset o t of the Gaussian point in the non-key scene representation model at the current moment, that is, o t =o t-1 +Δo t .

[0097] Step S34, according to the anchor point coordinates, the decoded feature residual field is interpolated to obtain each feature residual representation feature vector, and the feature residual representation feature vector is input into the decoded MLP to predict the feature residual between the current feature vector and the feature vector in the scene representation model at the previous moment through the MLP.

[0098] It should be noted that the feature residual representing feature vector refers to a vector used to describe the change characteristics of the feature vector, which contains the information of the feature vector residual.

[0099] In addition, it should be noted that in the embodiment, the motion field and the feature residual field are respectively modeled based on the transformation cache, and different training methods can be adopted for different three-dimensional dynamic scenes. Please refer to Figure 2 , a transformation cache is used to model the motion field and the feature residual field, that is, the anchor point coordinates x are used to interpolate the binary hash table A to obtain the corresponding feature vector B, and the feature vector is input into the MLP to predict the offset change Δo and the feature residual Δf, wherein Δo constitutes the motion field M, and Δf constitutes the feature residual field F; two or more transformation caches are used to model the motion and the feature residual, and for non-critical scenes, the transformation cache model of the current time is obtained through two-stage or even multi-stage training. Please refer to Figure 3 , in the first stage, only the transformation cache representing the Gaussian point offset change is learned, and in the second stage, the motion transformation cache parameters of the first stage are fixed, and the transformation cache model representing the feature residual is learned, that is, the anchor point coordinates x are used to interpolate the binary hash table A to obtain the corresponding feature vector B, and the feature vector is input into the MLP to predict the offset change Δo or the feature residual Δf, wherein Δo constitutes the motion field M, and Δf constitutes the feature residual field F.

[0100] It can be understood that since the change of the features in the scene needs to be described, step S33 is performed, which can avoid complete reconstruction of each feature vector, reduce unnecessary data processing operations, improve the efficiency of data processing, and at the same time maintain the details of scene reconstruction.

[0101] Exemplarily, according to the anchor point coordinates x, the preset feature residual field F t in the preset hash table is interpolated to obtain the feature residual representing feature vector interp(x, F t ), and the feature residual representing feature vector is input into the MLP to predict the feature residual Δf t of the feature vector:

[0102] Δf t = MLP2(interp(x, F t ))

[0103] wherein the feature residual Δf t is used to decode and combine the corresponding feature vector f t-1 in the scene representation model of the previous time to obtain the feature vector f t of the non-critical scene representation model of the current time, that is, f t= f t-1 + Δf t .

[0104] Step S35, combine each offset change with the corresponding Gaussian point position offset in the scene representation model at the previous time, and combine each feature residual with the corresponding feature vector in the scene representation model at the previous time, to obtain the non-key scene representation model.

[0105] Exemplarily, combine each offset change Δo t with the corresponding Gaussian point position offset o t-1 in the scene representation model at the previous time, to obtain the Gaussian point position offset o t of the non-key scene representation model at the current time, i.e. t = o t-1 + Δo t ; combine each feature residual Δf t with the corresponding feature vector f t-1 in the scene representation model at the previous time, to obtain the feature vector f t of the non-key scene representation model at the current time, i.e. t = f t-1 + Δf t . Based on the Gaussian point position offset o t , the feature vector f t , and the anchor point coordinates and scale at the previous time, the non-key scene representation model is obtained.

[0106] In this embodiment, by performing anchor point coordinate positioning, interpolation operation and neural network prediction, the high complexity and low efficiency problems caused by directly calculating all point motion and feature change are avoided, efficient reconstruction and accurate representation of dynamic scenes are realized, scene continuity and detail preservation are ensured, and data processing flow is optimized, reducing the computational burden and data volume.

[0107] Step S40, after traversing each non-key multi-view image, generate a three-dimensional volume video based on the key scene representation model and each non-key scene representation model.

[0108] It can be understood that, in order to synthesize a continuous three-dimensional volume video, step S40 is performed to provide an effective generation method for generating a three-dimensional volume video.

[0109] Exemplarily, after traversing each non-key multi-view image, the rendering of the virtual viewpoint is performed on each non-key scene representation model corresponding to each non-key multi-view image, and the rendering of the virtual viewpoint is also performed on the key scene representation model corresponding to the key multi-view image, so as to obtain the three-dimensional volume video finally needed to be obtained, so that the user can independently select a viewing angle and a playing time through dragging, clicking and other operations by using a player with rotation, translation, roll, zoom and other functions, thereby realizing the free interaction of the three-dimensional volume video.

[0110] The embodiment provides a three-dimensional volume video encoding and generating method, which realizes the fast reconstruction of the non-key scene by combining the preset transformation cache with the scene representation model at the last time during the reconstruction of the non-key scene, thereby avoiding the problems of high computational complexity and slow scene reconstruction speed caused by complete modeling and compression of the scene at each time in the traditional volume video generating method, realizing the effect of fast, accurate and high-quality generation of the three-dimensional volume video, providing a realistic immersive experience for the user, significantly improving the data processing efficiency of the volume video generation process, and reducing the data transmission amount after encoding.

[0111] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be described in detail hereinafter. On this basis, please refer to Figure 5 , the preset context model at least includes a hash grid and a multi-layer perception machine (MLP), and the three-dimensional volume video encoding and generating method further includes steps S201-S202:

[0112] In step S201, the first distortion loss and the first code rate loss in the key scene reconstruction process are calculated, and a first joint loss is constructed based on the first distortion loss and the first code rate loss.

[0113] It should be noted that the first distortion loss refers to the difference between the reconstructed scene and the original scene caused by quantization or encoding during the key scene reconstruction process, which reflects the decrease of the reconstruction quality; the first code rate loss refers to the amount of data consumed for transmitting or storing the reconstructed scene information during the encoding process, which reflects the loss of encoding efficiency; and the first joint loss is a comprehensive index combining the distortion loss and the code rate loss, which is used to balance the relationship between the reconstruction quality and the encoding efficiency in the optimization process.

[0114] Understandably, traditional methods often fail to consider the storage requirements of the model during scene reconstruction, and the large model size poses a challenge to the transmission of volumetric videos, affecting their practical application. Therefore, step S201, by simultaneously considering reconstruction quality (i.e., the first distortion loss) and data transmission cost (i.e., the first bitrate loss), avoids the problem of low usability of volumetric video generation methods caused by simply pursuing reconstruction quality. This provides an effective optimization and supervision method for achieving faster and more stable generation of volumetric videos without losing too much image quality.

[0115] For example, the first distortion loss D in the critical scene reconstruction process is calculated. (0) and the first bit rate loss R (0) The first joint loss L is calculated using the following formula. (0) :

[0116] L (0) =D (0) +λR (0)

[0117] Where λ represents the weighting coefficient of the first bit rate loss.

[0118] In one feasible implementation, the step of calculating the first distortion loss and the first bit rate loss in the key scene reconstruction process in step S201 may include steps S210 to S220:

[0119] Step S210: Calculate the first distortion loss based on the key multi-view image and the key scene representation model;

[0120] For example, the first distortion loss D is calculated using the following formula. (0) :

[0121] D (0) =(1-λ) ssim )×L1+λ ssim ×(1-SSIM)

[0122] Where, λ ssim The weights used to balance L1 loss and SSIM loss are L1, which is the absolute error, and SSIM (Structural Similarity Index Measure), which is a parameter determined based on the similarity between key multi-view images and key scene representation models.

[0123] Step S220: Calculate the first bitrate loss based on the probability distribution of the anchor point attributes, wherein the probability distribution is predicted by the preset scene compression model.

[0124] Exemplarily, the first code rate loss R is calculated by using the following formula (0) :

[0125]

[0126] wherein s represents the anchor point attribute, including a feature vector f, a size l and a Gaussian point position offset o, is the quantized anchor point attribute, represents a probability distribution of the anchor point attribute.

[0127] In the embodiment, by calculating the first distortion loss and the first code rate loss, an effective data basis is provided for comprehensively considering the reconstruction quality and the data transmission cost.

[0128] In step S202, the anchor point attribute and the scene compression model are optimized according to the first joint loss, to obtain an adjusted anchor point attribute and a trained scene compression model, and based on the adjusted anchor point attribute and the trained scene compression model, the step of quantizing each anchor point attribute by using the preset scene compression model is performed.

[0129] It can be understood that, since the process of quantizing and compressing the anchor point attribute is often difficult to directly achieve optimal performance, step S202 is performed to optimize the anchor point attribute and the scene compression model, so as to improve the accuracy of the reconstruction algorithm, reduce the reconstruction error, and improve the coding efficiency, thereby realizing a three-dimensional volumetric video reconstruction with higher quality and higher efficiency.

[0130] Exemplarily, in the case where the scene compression model is a context model (including a hash grid and an MLP), by continuously adjusting each anchor point attribute, each parameter in the hash grid in the context model, and the weights and biases of each layer neural network in the MLP, the joint loss is minimized, so that each anchor point attribute, each parameter in the hash grid in the context model, and the weights and biases of each layer neural network in the MLP, which minimize the joint loss, are used as the adjusted anchor point attribute and the trained context model, and the quantization step and the probability distribution of the adjusted anchor point attribute are predicted based on the trained context model.

[0131] In the embodiment, by jointly optimizing the anchor point attribute and the scene compression model, the problems of quality degradation and low data transmission efficiency caused by separately optimizing distortion or code rate in the process of three-dimensional volumetric video reconstruction are avoided, the data compression efficiency is improved while the quality of the reconstructed scene is ensured, so that a balance between high quality and high efficiency coding is achieved, and the performance of the overall video reconstruction system is improved.

[0132] Based on the first embodiment and / or the third embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above-mentioned embodiment one and embodiment two can be referred to the above description, and the subsequent will not be described in detail. On this basis, please refer to Figure 6 , the three-dimensional volume video encoding and generating method further includes steps S301-S304:

[0133] Step S301, calculate the second distortion loss and the second code rate loss in the non-key scene reconstruction process, and construct a second joint loss based on the second distortion loss and the second code rate loss;

[0134] It should be noted that the second distortion loss refers to the difference between the reconstructed scene and the original scene in the non-key scene reconstruction process, which measures the loss of reconstruction quality; the second code rate loss refers to the amount of data consumed for encoding the non-key scene information, which reflects the efficiency loss of data transmission or storage in the encoding process; the second joint loss is a composite loss function combining the distortion loss and the code rate loss, used to balance the reconstruction quality and the encoding efficiency in the optimization process.

[0135] It can be understood that, since the traditional method often fails to consider the storage occupation of the model in the scene reconstruction process, and the large model size brings challenges to the transmission of volume video, affecting the practical application of volume video, step S301 is performed to consider the reconstruction quality (i.e. the second distortion loss) and the data transmission cost (i.e. the second code rate loss) at the same time, which can avoid the problem that the simplicity of pursuing reconstruction quality leads to low availability of the volume video generation method, and further provides an effective optimization supervision method for further reducing the model size without losing too much quality, realizing faster and more stable generation of volume video.

[0136] Exemplarily, the second distortion loss D (t) and the second code rate loss R (t) in the non-key scene reconstruction process are calculated, and the second joint loss L (t) is calculated by the following formula:

[0137] L (t) = D (t) + λR (t)

[0138] Wherein, λ represents the weight coefficient of the second code rate loss.

[0139] In one possible implementation, the step of calculating the second distortion loss and the second code rate loss in the non-key scene reconstruction process in step S301 can include steps S310-S320:

[0140] Step S310, the second distortion loss is calculated according to the non-key multi-view image and the non-key scene representation model.

[0141] Exemplarily, the second distortion loss D is calculated by the following formula: (t) :

[0142] D (t) =(1-λ ssim )×L1+λ ssim ×(1-SSIM)

[0143] Wherein, λ ssim is a weight for balancing the L1 loss and the SSIM loss, L1 is an absolute error, and SSIM is a parameter determined based on the similarity between the non-key multi-view image and the non-key scene representation model.

[0144] Step S320, the second code rate loss is calculated according to the existing number and the existing frequency of the binary numbers in the preset motion field and the preset feature residual field.

[0145] Exemplarily, the second code rate loss R is calculated by the following formula: (t) :

[0146] R (t) =M + ×(-log2(h f ))+M - ×(-log2(1-h f ))

[0147] Wherein, M + and M - respectively represent the binary multi-level hash grids storing the motion field and the feature residual field, that is, the number of “+1” and “-1” in the preset hash table, and h f represents the appearance frequency of the “+1” symbol, that is, the existing frequency.

[0148] In the embodiment, by calculating the second distortion loss and the second code rate loss, an effective data basis is provided for comprehensively considering the reconstruction quality and the data transmission cost.

[0149] Step S302, the preset motion field, the preset feature residual field and the MLP are optimized according to the second joint loss, to obtain an optimized motion field, an optimized feature residual field and a trained MLP.

[0150] It can be understood that, since the preset motion field, feature residual field and MLP cannot be well adapted to the reconstruction requirements of non-key scenes, step S302 is performed to avoid the reconstruction error caused by the mismatch of the motion field, feature residual field and MLP, thereby improving the accuracy and robustness of the non-key scene reconstruction.

[0151] Exemplarily, by continuously adjusting the parameters of the motion field and the feature residual field cached in the binary multi-level hash grid, and the weights and biases of the neural networks in each layer of the MLP, the joint loss is minimized, so that the motion field, the feature residual field and the MLP that minimize the joint loss are taken as the optimized motion field and feature residual field, and the trained MLP.

[0152] Step S303, based on the optimized motion field and the trained MLP, returns to perform the step of interpolating the decoded motion field according to the anchor point coordinates;

[0153] Exemplarily, the optimized motion field is three-dimensionally linearly interpolated according to the anchor point coordinates to obtain each motion representation feature vector, and each motion representation feature vector is input into the trained MLP to more accurately predict the displacement change between the current time instant each Gaussian point position offset and the each Gaussian point position offset in the scene representation model at the previous time instant.

[0154] Step S304, based on the optimized feature residual field and the trained MLP, returns to perform the step of interpolating the decoded feature residual field according to the anchor point coordinates.

[0155] Exemplarily, the optimized feature residual field is three-dimensionally linearly interpolated according to the anchor point coordinates to obtain each feature residual representation feature vector, and each feature residual representation feature vector is input into the trained MLP to more accurately predict the feature residual between the current each feature vector and each feature vector in the scene representation model at the previous time instant.

[0156] In this embodiment, by jointly optimizing the motion field, the feature residual field and the MLP, the distortion problem in non-key scene reconstruction, low encoding efficiency and waste of computing resources are avoided, and the encoding efficiency and the computing complexity are improved while the reconstruction quality is ensured, so that the non-key scene reconstruction process is optimized to improve the performance of the overall three-dimensional volume video encoding and generation system.

[0157] Exemplarily, in order to help understand the implementation process of the three-dimensional volume video encoding and generation method obtained after combining the above-mentioned embodiment one, please refer to Figure 7 , Figure 7A brief flowchart of a three-dimensional volumetric video encoding and generation method is provided, specifically:

[0158] The multi-view image corresponding to the time t=0 (the starting time) is the key multi-view image, and the multi-view images corresponding to the subsequent times t=1, t=2, and the like are non-key multi-view images. For the key multi-view image, first, key scene reconstruction is performed in the encoder to obtain a three-dimensional Gaussian representation of the key scene, and the key scene is entropy encoded to obtain a binary code stream of the key scene. In the decoder, the binary code stream of the key scene is decoded to obtain a key scene representation model. For the non-key multi-view image at any subsequent time, the preset transformation buffer is encoded to obtain a binary code stream of the non-key scene. In the decoder, the binary code stream of the non-key scene is decoded, and the scene representation model at the previous time is combined to obtain a non-key scene representation model. Finally, the scene representation models are rendered to obtain a three-dimensional volumetric video for user interaction, such as camera pose adjustment and playback time control.

[0159] Further, please refer to Figure 8 , Figure 8 An encoding framework diagram of a three-dimensional volumetric video encoding and generation method is provided, specifically:

[0160] For the key multi-view image at t=0, camera calibration is performed to obtain a sparse point cloud, and an anchor-based Gaussian representation model is initialized based on the sparse point cloud to obtain a key scene. The key scene is quantized Q, entropy encoded, and dequantized DQ through a context model to obtain a decoded key scene representation model S0. For the non-key multi-view image at t=1, based on the transformation anchor parameters predicted from the hash table in the binary transformation buffer, the key scene representation model is combined to obtain a non-key scene representation model S1 through a transformation anchor parameter operation T. The operation at t=1 is repeated for subsequent times t=i. In this application, an anchor-based Gaussian representation method is used, that is, the anchor properties of the scene representation model S i , including anchor coordinates x, feature vectors f, size f, and Gaussian point position offset o, are processed through an MLP to output a view A represented by three-dimensional Gaussian: opacity alpha, three primary color information c, scale s, quaternion r, and mean value mu.

[0161] The application compares and analyzes the N3DV dataset with the existing volume video online reconstruction methods StreamRF and 3DGStream, and the objective results are shown in Table 1 below, wherein PSNR represents the peak signal-to-noise ratio, Storage represents the storage space occupied, Train represents the training time, and Render represents the video frame rate after rendering. The experimental results show that the application realizes the highest rendering quality with the lowest storage cost, and the training time is similar to the StreamRF and 3DGStream methods, supporting real-time rendering (FPS> 30). In the above experiment, a one-stage training method is used, and a transformation cache model is used to represent the Gaussian point position offset and feature residual at the same time. The binary hash grid is realized based on three planes.

[0162] Method PSNR (dB) Storage (MB) Train (mins) Render (FPS) Stream RF 30.68 31.4 0.23 8.3 3DG Stream 31.67 7.8 0.20 215 The present application 32.32 0.09 0.22 126

[0163] Table 1

[0164] It should be noted that the above examples are only used for understanding the application and do not constitute a limitation on the three-dimensional volume video coding and generation method of the application. More simple transformations based on this technical concept are within the protection scope of the application.

[0165] The application also provides a three-dimensional volume video coding and generation device, which is described in detail below with reference to Figure 9 The three-dimensional volume video coding and generation device comprises:

[0166] The acquisition module 10 is configured to acquire a multi-view video captured by a plurality of cameras, and determine a key multi-view image at a starting time and a plurality of non-key multi-view images at subsequent times in the multi-view video.

[0167] The first encoding module 20 is configured to reconstruct a key scene of the multi-view video according to the key multi-view image, and encode the key scene to obtain a code stream of the key scene, so as to obtain a key scene representation model of the multi-view video by decoding.

[0168] The second encoding module 30 is configured to, for any non-key multi-view image, encode a preset transformation cache to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to obtain a non-key scene representation model of the multi-view video by decoding and combining a scene representation model at a previous time.

[0169] The generation module 40 is configured to generate a three-dimensional volume video based on the key scene representation model and the non-key scene representation models after traversing all the non-key multi-view images.

[0170] Optionally, the first encoding module 20 is further configured to:

[0171] obtaining anchor point attributes of each anchor point in the key scene;

[0172] quantizing each anchor point attribute by using a preset scene compression model, and performing entropy coding on the quantized anchor point attributes to obtain a code stream of the key scene.

[0173] Optionally, the first optimization module 50 in the three-dimensional volumetric video coding and generation apparatus is configured to:

[0174] calculate a first distortion loss and a first code rate loss in the key scene reconstruction process, and construct a first joint loss based on the first distortion loss and the first code rate loss;

[0175] optimize the anchor point attributes and the scene compression model according to the first joint loss to obtain adjusted anchor point attributes and a trained scene compression model, and perform the quantizing each anchor point attribute by using a preset scene compression model based on the adjusted anchor point attributes and the trained scene compression model.

[0176] Optionally, the first optimization module 50 is further configured to:

[0177] calculate the first distortion loss according to the key multi-view image and the key scene representation model;

[0178] calculate the first code rate loss according to a probability distribution of the anchor point attributes, wherein the probability distribution is predicted by the preset scene compression model.

[0179] Optionally, the preset transformation cache includes a preset motion field, a preset feature residual field, and a preset multi-layer perception (MLP), and the second encoding module 30 is further configured to:

[0180] encode the preset motion field, the preset feature residual field, and the preset MLP to obtain a code stream of the non-key scene;

[0181] when decoding the code stream of the non-key scene, obtain anchor point coordinates of each anchor point in a scene corresponding to a multi-view image at a previous time point;

[0182] interpolate the decoded motion field according to the anchor point coordinates to obtain each motion representation feature vector, and input the each motion representation feature vector into the decoded MLP to predict a change in offset between each Gaussian point position offset at a current time point and each Gaussian point position offset in a scene representation model at a previous time point through the MLP;

[0183] interpolating the decoded motion field according to the anchor point coordinates to obtain a plurality of feature residual vectors, and inputting the plurality of feature residual vectors into the decoded MLP to predict feature residuals between the current feature vectors and the feature vectors in the previous scene representation model through the MLP;

[0184] combining the plurality of offset changes with the corresponding offset of the Gaussian point in the previous scene representation model, and combining the plurality of feature residuals with the corresponding feature vectors in the previous scene representation model to obtain the non-key scene representation model.

[0185] Optionally, the second optimization module 60 in the three-dimensional volumetric video coding and generation apparatus is configured to:

[0186] calculate a second distortion loss and a second code rate loss in the non-key scene reconstruction process, and construct a second joint loss based on the second distortion loss and the second code rate loss;

[0187] optimize the preset motion field, the preset feature residual field and the MLP according to the second joint loss to obtain an optimized motion field, an optimized feature residual field and a trained MLP;

[0188] based on the optimized motion field and the trained MLP, return to the step of interpolating the decoded motion field according to the anchor point coordinates;

[0189] based on the optimized feature residual field and the trained MLP, return to the step of interpolating the decoded feature residual field according to the anchor point coordinates.

[0190] Optionally, the second optimization module 60 is further configured to:

[0191] calculate the second distortion loss according to the non-key multi-view image and the non-key scene representation model;

[0192] calculate the second code rate loss according to the number and frequency of the existing binary numbers in the preset motion field and the preset feature residual field.

[0193] The three-dimensional volumetric video coding and generation apparatus provided in the present application adopts the three-dimensional volumetric video coding and generation method in the above embodiments, and can solve the technical problem of how to efficiently code and generate a three-dimensional volumetric video. Compared with the prior art, the three-dimensional volumetric video coding and generation apparatus provided in the present application has the same beneficial effects as the three-dimensional volumetric video coding and generation method provided in the above embodiments, and other technical features in the three-dimensional volumetric video coding and generation apparatus are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0194] The application provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional volume video coding and generation method in the above embodiment one.

[0195] Reference will be made to the following description Figure 10 , which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the application. The electronic device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the application.

[0196] As Figure 10As shown, the electronic device can include a processing device 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0197] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0198] The electronic device provided by the present application adopts the three-dimensional volume video coding and generation method in the above-mentioned embodiments, which can solve the technical problem of how to efficiently code and generate three-dimensional volume video. Compared with the prior art, the electronic device provided by the present application has the same beneficial effects as the three-dimensional volume video coding and generation method provided by the above-mentioned embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0199] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0200] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to perform the three-dimensional volume video coding and generation method in the above embodiments.

[0201] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination of the above.

[0202] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: acquire a multi-view video photographed by a multi-camera, and determine a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video; reconstruct a key scene of the multi-view video according to the key multi-view image, and encode the key scene to obtain a code stream of the key scene, so as to be decoded to obtain a key scene representation model of the multi-view video; for any non-key multi-view image, encode a preset transformation cache to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to be decoded to combine a scene representation model at a previous time to obtain a non-key scene representation model of the multi-view video; after traversing each non-key multi-view image, generate a three-dimensional volume video based on the key scene representation model and each non-key scene representation model.

[0203] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0204] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0205] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the three-dimensional volume video encoding and generation method described above, and can solve the technical problem of how to efficiently encode and generate a three-dimensional volume video. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the three-dimensional volume video encoding and generation method provided by the above-mentioned embodiments, which will not be repeated here.

[0206] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the three-dimensional volumetric video encoding and generating method as described above.

[0207] The computer program product provided by the application can solve the technical problem of how to efficiently encode and generate a three-dimensional volumetric video. Compared with the prior art, the computer program product provided by the application has the same beneficial effects as the three-dimensional volumetric video encoding and generating method provided by the above-mentioned embodiments, which will not be described here.

Claims

1. A method for encoding and generating a three-dimensional volume video, characterized by, The three-dimensional volumetric video encoding and generating method comprises: acquiring a multi-view video captured by multiple cameras, and determining a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video; reconstructing a key scene of the multi-view video according to the key multi-view image, and acquiring anchor point attributes of each anchor point in the key scene; quantizing each anchor point attribute through a preset scene compression model, and entropy encoding each quantized anchor point attribute to obtain a code stream of the key scene, so as to decode to obtain a key scene representation model of the multi-view video; for any non-key multi-view image, encoding a preset transformation cache to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to combine the scene representation model at the previous time after decoding to obtain a non-key scene representation model of the multi-view video; generating a three-dimensional volumetric video based on the key scene representation model and the non-key scene representation models after traversing each non-key multi-view image; the preset transformation cache comprises a preset motion field, a preset feature residual field and a preset multi-layer perception (MLP), and the step of encoding the preset transformation cache to obtain the code stream of the non-key scene corresponding to the non-key multi-view image, so as to combine the scene representation model at the previous time after decoding to obtain the non-key scene representation model of the multi-view video comprises: encoding the preset motion field, the preset feature residual field and the preset MLP to obtain the code stream of the non-key scene; when decoding the code stream of the non-key scene, acquiring anchor point coordinates of each anchor point in the scene corresponding to the multi-view image at the previous time; interpolating the decoded motion field according to the anchor point coordinates to obtain each motion feature vector, and inputting the each motion feature vector into the decoded MLP to predict a change in offset between the position offset of each Gaussian point at the current time and the position offset of each Gaussian point in the scene representation model at the previous time through the MLP; interpolating the decoded feature residual field according to the anchor point coordinates to obtain each feature residual feature vector, and inputting the each feature residual feature vector into the decoded MLP to predict a feature residual between the current feature vector and the feature vector in the scene representation model at the previous time through the MLP; combining each change in offset with the corresponding position offset of the Gaussian point in the scene representation model at the previous time, and combining each feature residual with the corresponding feature vector in the scene representation model at the previous time to obtain the non-key scene representation model.

2. The method of claim 1, wherein the three-dimensional volume video is encoded and generated by: The three-dimensional volumetric video encoding and generating method further comprises: calculating a first distortion loss and a first code rate loss in the key scene reconstruction process, and constructing a first joint loss based on the first distortion loss and the first code rate loss; According to the first joint loss, the anchor point attribute and the scene compression model are optimized to obtain an adjusted anchor point attribute and a trained scene compression model, and based on the adjusted anchor point attribute and the trained scene compression model, the step of quantizing each anchor point attribute by using the preset scene compression model is performed.

3. The method of claim 2, wherein the three-dimensional volume video is encoded and generated by: The step of calculating the first distortion loss and the first code rate loss in the key scene reconstruction process comprises: According to the key multi-view image and the key scene representation model, the first distortion loss is calculated. According to the probability distribution of the anchor point attribute, the first code rate loss is calculated, wherein the probability distribution is predicted by the preset scene compression model.

4. The method of claim 1, wherein the three-dimensional volume video encoding and generating method is characterized by, The three-dimensional volumetric video coding and generation method further comprises: calculating a second distortion loss and a second code rate loss in a non-key scene reconstruction process, and constructing a second joint loss based on the second distortion loss and the second code rate loss; According to the second joint loss, the preset motion field, the preset feature residual field and the MLP are optimized to obtain an optimized motion field, an optimized feature residual field and a trained MLP; Based on the optimized motion field and the trained MLP, the step of interpolating the decoded motion field according to each anchor point coordinate is returned to perform; Based on the optimized feature residual field and the trained MLP, the step of interpolating the decoded feature residual field according to each anchor point coordinate is returned to perform.

5. The method of claim 4, wherein the three-dimensional volume video is encoded and generated by: The step of calculating the second distortion loss and the second code rate loss in the non-key scene reconstruction process comprises: According to the non-key multi-view image and the non-key scene representation model, the second distortion loss is calculated. According to the number and frequency of the existence of binary numbers in the preset motion field and the preset feature residual field, the second code rate loss is calculated.

6. A three-dimensional volumetric video encoding and generating apparatus, characterized in that, The three-dimensional volumetric video coding and generation apparatus comprises: An acquisition module is configured to acquire a multi-view video captured by multiple cameras, and determine a key multi-view image at a starting time and non-key multi-view images at subsequent times in the multi-view video; A first coding module is configured to reconstruct a key scene of the multi-view video according to the key multi-view image, and acquire anchor point attributes of each anchor point in the key scene; quantize each anchor point attribute by using a preset scene compression model, and entropy encode the quantized anchor point attributes to obtain a code stream of the key scene, so as to decode a key scene representation model of the multi-view video; A second key module is configured to, for any non-key multi-view image, encode a preset transformation buffer to obtain a code stream of a non-key scene corresponding to the non-key multi-view image, so as to combine a scene representation model at a previous time after decoding to obtain a non-key scene representation model of the multi-view video; A generation module is configured to, after traversing each non-key multi-view image, generate a three-dimensional volumetric video based on the key scene representation model and each non-key scene representation model. The preset transform cache includes a preset motion field, a preset feature residual field and a preset multi-layer perceptron (MLP), and the second key module is further configured to encode the preset motion field, the preset feature residual field and the preset multi-layer perceptron (MLP) to obtain a bitstream of the non-key scene; when the bitstream of the non-key scene is decoded, anchor point coordinates of each anchor point in a scene corresponding to a multi-view image at a previous time are obtained; each motion representation feature vector is obtained by interpolating the decoded motion field according to the anchor point coordinates, and the each motion representation feature vector is input into the decoded MLP to predict a change in offset between a position offset of each Gaussian point at a current time and a position offset of each Gaussian point in a scene representation model at a previous time; each feature residual representation feature vector is obtained by interpolating the decoded feature residual field according to the anchor point coordinates, and the each feature residual representation feature vector is input into the decoded MLP to predict a feature residual between a current feature vector and each feature vector in the scene representation model at the previous time; and the non-key scene representation model is obtained by combining each change in offset and the corresponding position offset of the Gaussian point in the scene representation model at the previous time, and combining each feature residual and the corresponding feature vector in the scene representation model at the previous time.

7. An electronic device, comprising: The device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the three-dimensional volumetric video encoding and generation method according to any one of claims 1 to 5.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the three-dimensional volumetric video encoding and generation method according to any one of claims 1 to 5.

Images