Three-dimensional model generation method and apparatus, device, storage medium, and program product

Abstract

A three-dimensional model generation method and apparatus, a device, a storage medium, and a program product, relating to the technical field of computers. The method comprises: acquiring a noise addition feature representation (210); when denoising the noise addition feature representation by means of denoising network layers respectively corresponding to a plurality of viewing angles, determining input feature representations of the denoising network layers respectively corresponding to the plurality of viewing angles (220); acquiring three-dimensional shared information shared among three-dimensional conversion matrices respectively corresponding to the plurality of input feature representations (230); on the basis of the three-dimensional shared information, adjusting the plurality of input feature representations (240); and on the basis of a plurality of adjusted feature representations, respectively generating viewing angle images at the plurality of viewing angles (250).

Description

Three-dimensional model generation method, device, equipment, storage medium and program product

[0001] This application claims priority to Chinese patent application number 202311205626.1, filed on September 15, 2023, entitled “Three-dimensional model generation method, device, equipment, storage medium and program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer technology, and in particular to a three-dimensional model generation method, apparatus, device, storage medium, and program product. Background Art

[0003] Multi-view learning, also known as multi-perspective learning, focuses on how to model and explore two-dimensional images collected from entities to obtain three-dimensional models that can better display the three-dimensional shape of the entity.

[0004] In related technologies, when a three-dimensional model is obtained based on two-dimensional image modeling, a diffusion model is usually used as a generation model. After obtaining an image captured for an entity, the image is converted into an abstract description of the entity through a vision language (Vision-Language) model. When the generation model is used to analyze the camera parameters corresponding to the image and other perspectives, the abstract description is used as the generation condition, so that the diffusion model can generate a three-dimensional model with a 360° perspective under the constraints of the generation conditions.

[0005] In these methods, the generation of 3D models relies solely on an abstract description of the image, which makes it difficult to guarantee geometric consistency. For example, if the input is a car image, the Vision-Language model generates an abstract description of the car's style, color, and appearance. However, the resulting multi-view images typically only satisfy this abstract description and cannot guarantee that they depict the same car.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a method, apparatus, device, storage medium, and program product for generating a three-dimensional model. The technical solution is as follows.

[0008] In one aspect, a three-dimensional model generation method is provided, which is executed by a computer device, and the method includes:

[0009] Obtaining a noisy feature representation corresponding to the noise data, wherein the noisy feature representation is used for denoising at multiple viewing angles to obtain viewing angle images corresponding to the entity elements at the multiple viewing angles;

[0010] When denoising the noisy feature representation through the denoising network layers corresponding to the multiple perspectives, respectively, determining input feature representations of the denoising network layers corresponding to the multiple perspectives, the input feature representations being feature representations input to the denoising network layers;

[0011] Obtaining three-dimensional shared information shared by three-dimensional transformation matrices corresponding to a plurality of input feature representations, wherein the three-dimensional transformation matrix is ​​a matrix obtained by transforming the dimensions of the input feature representations;

[0012] adjusting the plurality of input feature representations based on the three-dimensional shared information to obtain a plurality of adjusted feature representations, wherein a corresponding relationship exists between the plurality of input feature representations and the plurality of adjusted feature representations;

[0013] Based on the multiple adjustment feature representations, perspective images corresponding to the entity element at the multiple perspectives are generated, and the multiple perspective images are used to integrate and generate a three-dimensional model representing the entity element.

[0014] In another aspect, a three-dimensional model generation device is provided, the device comprising:

[0015] An acquisition module, configured to acquire a noise feature representation corresponding to the noise data, wherein the noise feature representation is used to remove noise from multiple viewing angles to obtain viewing angle images corresponding to the entity elements at the multiple viewing angles;

[0016] a determination module, configured to determine, when denoising the noisy feature representation through the denoising network layers corresponding to the multiple perspectives, input feature representations of the denoising network layers corresponding to the multiple perspectives, respectively, the input feature representations being feature representations input to the denoising network layers;

[0017] an extraction module, configured to obtain three-dimensional shared information shared by three-dimensional transformation matrices corresponding to a plurality of input feature representations, wherein the three-dimensional transformation matrix is ​​a matrix obtained by transforming the dimensions of the input feature representations;

[0018] an adjustment module, configured to adjust the plurality of input feature representations based on the three-dimensional shared information to obtain a plurality of adjusted feature representations, wherein a corresponding relationship exists between the plurality of input feature representations and the plurality of adjusted feature representations;

[0019] A generation module is used to generate perspective images corresponding to the entity element under the multiple perspectives based on the multiple adjustment feature representations, and the multiple perspective images are used to integrate and generate a three-dimensional model representing the entity element.

[0020] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the three-dimensional model generation method as described in any of the above-mentioned embodiments of the present application.

[0021] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a three-dimensional model generation method as described in any of the above-mentioned embodiments of the present application.

[0022] In another aspect, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the three-dimensional model generation method described in any of the above embodiments.

[0023] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0024] By comprehensively analyzing multiple input feature representations, the same information of multiple input feature representations on a three-dimensional scale can be obtained, thereby extracting three-dimensional shared information. The input feature representation is constrained with the help of the three-dimensional shared information, thereby avoiding the problem of fragmentation when different perspectives are independently denoised through the corresponding denoising network layer. Adjusting the input feature representation through the three-dimensional shared information is conducive to strengthening the correlation in the denoising process of different perspectives, thereby making the perspective images have a stronger correlation, which is conducive to generating a more geometrically consistent three-dimensional model based on the integration of multiple perspective images, and improving the authenticity and meticulousness of the entity elements represented by the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0026] FIG2 is a flow chart of a method for generating a three-dimensional model provided by an exemplary embodiment of the present application;

[0027] FIG3 is a flow chart of a method for generating a three-dimensional model provided by another exemplary embodiment of the present application;

[0028] FIG4 is a flowchart of a method for generating a three-dimensional model provided by another exemplary embodiment of the present application;

[0029] FIG5 is a flowchart of a method for generating a three-dimensional model provided by another exemplary embodiment of the present application;

[0030] FIG6 is a schematic diagram of a denoising process performed by denoising autoencoders corresponding to two perspectives, provided by an exemplary embodiment of the present application;

[0031] FIG7 is a schematic diagram of the internal structure of the pooling layer of the attention mechanism provided by an exemplary embodiment of the present application;

[0032] FIG8 is a schematic diagram of a denoising autoencoder cyclically called by a traditional diffusion model provided by an exemplary embodiment of the present application;

[0033] FIG9 is a schematic diagram comparing the generation results of a three-dimensional model generation method provided by an exemplary embodiment of the present application with the generation results of other methods;

[0034] FIG10 is a schematic diagram comparing the generation results of a three-dimensional model generation method provided by another exemplary embodiment of the present application with the generation results of other methods;

[0035] FIG11 is a structural block diagram of a three-dimensional model generating apparatus provided by an exemplary embodiment of the present application;

[0036] FIG12 is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0037] In an embodiment of the present application, a method for generating a three-dimensional model is introduced. Through the method for generating a three-dimensional model, a denoising process is performed on the noisy feature representation at multiple perspectives, and three-dimensional shared information between the input feature representations is extracted during the denoising process, so that the input feature representation can be constrained with the help of the three-dimensional shared information, and the correlation in the denoising process of different perspectives is strengthened, so that there is a strong correlation between the perspective images, which is conducive to generating a three-dimensional model with stronger geometric consistency based on the integration of multiple perspective images. The above-mentioned three-dimensional model generation method can be applied to various three-dimensional model modeling scenarios such as game modeling, medical field (such as modeling medical research objects), film field, scientific field (such as modeling accurate models of compounds), architectural field, geological field, etc., and the embodiment of the present application does not limit this.

[0038] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant region. For example, the noise-added data, denoising network layer and other contents involved in this application are all obtained with full authorization.

[0039] Next, the 3D model generation system involved in the embodiments of this application is described. The 3D model generation method provided in the embodiments of this application can be implemented by a terminal alone, by a server, or by a terminal and a server through data interaction, and this is not limited in this embodiment of the application. Optionally, the 3D model generation method is described using the interaction between a terminal and a server as an example.

[0040] Schematically, please refer to FIG. 1 , the three-dimensional model generation system involves a terminal 110 and a server 120 , and the terminal 110 and the server 120 are connected via a communication network 130 .

[0041] In some embodiments, the terminal 110 has a noise generating function or a noise acquiring function for acquiring noise data.

[0042] Optionally, the terminal 110 sends noise data to the server 120 through the communication network 130. The server 120 can obtain a noisy feature representation based on the noise data. The noisy feature representation is used to denoise separately at multiple perspectives to obtain perspective images corresponding to the entity elements at multiple perspectives.

[0043] Schematically, the multiple perspectives are pre-selected different perspectives, and the noise-added feature indicates that the denoising process is performed separately at each perspective.

[0044] In some embodiments, when denoising the noisy feature representation using the denoising network layers corresponding to the multiple perspectives, the server 120 determines the input feature representations of the denoising network layers corresponding to the multiple perspectives, where the input feature representations are feature representations to be input to the denoising network layers for denoising.

[0045] Illustratively, each perspective corresponds to at least one denoising network layer, and there is a correspondence between the denoising network layers corresponding to each perspective, such as a correspondence between the first denoising network layer corresponding to perspective A and the first denoising network layer corresponding to perspective B. When the server 120 performs denoising at multiple perspectives based on the noisy feature representation, the noisy feature representation is decoded by at least one denoising network layer corresponding to each of the multiple perspectives. For any denoising network layer corresponding to each of the multiple perspectives, the feature representation input to the denoising network layer is determined as the input feature representation.

[0046] In some embodiments, the server 120 extracts three-dimensional shared information shared between the three-dimensional transformation matrices corresponding to the multiple input feature representations.

[0047] Among them, the three-dimensional transformation matrix is ​​obtained by dimensional transformation of the input feature representation.

[0048] Schematically, for any denoising network layer, there is a one-to-one correspondence between at least one denoising network layer corresponding to multiple perspectives, so it is possible to determine the input feature representations of the multiple perspectives in the corresponding denoising network layer, that is, to determine the input feature representations corresponding to the multiple perspectives.

[0049] Optionally, to analyze 3D information of a 3D entity element based on multiple 2D input feature representations, the multiple input feature representations are individually dimensionally transformed to obtain 3D transformation matrices corresponding to the multiple input feature representations. The multiple 3D transformation matrices are then analyzed to extract shared 3D information, i.e., 3D shared information that characterizes the elemental information content expressed by the multiple 2D input feature representations after conversion to 3D features.

[0050] In some embodiments, the server 120 adjusts the multiple input feature representations using the three-dimensional shared information to obtain multiple adjusted feature representations.

[0051] There is a corresponding relationship between multiple input feature representations and multiple adjustment feature representations.

[0052] Illustratively, 3D shared information can demonstrate the correlation between multiple input feature representations at a 3D scale, thereby constraining the decoding process of noisy feature representations from multiple viewpoints using this information. The 3D shared information is then applied to each of the multiple input feature representations to generate multiple adjusted feature representations, each of which exhibits good geometric consistency at a 3D scale.

[0053] In some embodiments, the server 120 generates perspective images corresponding to the entity element at multiple perspectives based on the multiple adjustment feature representations.

[0054] Among them, multiple perspective images are used to integrate and generate a three-dimensional model representing the entity elements.

[0055] Illustratively, the server 120 obtains a perspective image under a corresponding perspective based on each adjusted feature representation, wherein the perspective image is an image obtained by predicting the entity element through the perspective. Optionally, multiple perspective images can be integrated to generate a three-dimensional model representing the entity element.

[0056] In some embodiments, the server 120 sends rendering data for rendering and displaying the three-dimensional model to the terminal 110 via the communication network 130. Optionally, the terminal 110 renders and displays the three-dimensional model based on the rendering data.

[0057] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc., and can also be implemented as desktop computers, etc.; the above-mentioned servers can be independent physical servers, or server clusters or distributed systems composed of multiple physical servers, or cloud servers.

[0058] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0059] In combination with the above-mentioned terminology introduction and application scenarios, the three-dimensional model generation method provided in this application is explained. Taking the application of this method to a server as an example, as shown in FIG2 , the method includes the following steps 210 to 250 .

[0060] Step 210: Obtain a noise-added feature representation corresponding to the noise data.

[0061] Illustratively, the noise data is data used to represent noise. Optionally, the noise data is randomly selected noise.

[0062] The noisy feature representation is a feature representation used to characterize the data information of the noise data. Optionally, the noisy feature representation is obtained by sampling the noise data from a pre-acquired Gaussian distribution. Illustratively, the noise data is a pre-provided Gaussian noise image, and features of the Gaussian noise image are extracted to obtain the noisy feature representation.

[0063] The noise feature representation is used to remove noise from multiple perspectives to obtain perspective images corresponding to the entity elements at multiple perspectives.

[0064] Schematically, a physical element is a three-dimensional solid element that is required for 3D model synthesis. For example, a physical element is an object element existing in the real world, such as a building, a tree, or a book; or a physical element is an object element synthesized in the virtual world, such as a virtual character or a virtual building. Schematically, multiple perspectives are different perspectives, and perspective is used to represent the angle of observation used when observing a physical element.

[0065] Optionally, the multiple perspectives are pre-selected. For example, the multiple perspectives include a front perspective for displaying a front view of a physical element, a side perspective for displaying a side view of a physical element, and a top perspective for displaying a view of a physical element. Illustratively, the multiple perspectives can also be represented in the form of different spatial angles.

[0066] In some embodiments, each of the multiple perspectives corresponds to a denoising model, and the denoising model is a model trained based on the perspective. For example, for denoising model 1 corresponding to perspective A, denoising model 1 is a model trained based on multiple sample images collected from perspective A.

[0067] When denoising the noisy feature representation at multiple perspectives, the noisy feature representation is input into the denoising models corresponding to the multiple perspectives. Each denoising model denoises the noisy feature representation based on the corresponding perspective, thereby obtaining a perspective image that represents the expected observation of the entity element at that perspective.

[0068] For example: the noisy feature representation is input into the denoising model 1 corresponding to the perspective A, and the noisy feature representation is denoised by the denoising model 1, and the perspective image a when the entity element is observed from the perspective A is predicted; in addition, the noisy feature representation is input into the denoising model 2 corresponding to the perspective B, and the noisy feature representation is denoised by the denoising model 2, and the perspective image b when the entity element is observed from the perspective B is predicted, etc.

[0069] Step 220 , when denoising the noisy feature representation through the denoising network layers corresponding to the multiple perspectives, determining the input feature representations of the denoising network layers corresponding to the multiple perspectives.

[0070] Optionally, for any one of the multiple perspectives, the perspective corresponds to at least one denoising network layer, where the denoising network layer is a network layer used to perform a denoising process. Illustratively, the denoising models corresponding to the multiple perspectives respectively include at least one denoising network layer. After the noisy feature representation is input into the denoising model corresponding to each perspective, the denoising process is performed on the noisy feature representation via the at least one denoising network layer.

[0071] Schematically, multiple perspectives correspond to multiple denoising network layers, and there is a correspondence between the multiple denoising network layers corresponding to the multiple perspectives. For example, the denoising models corresponding to the multiple perspectives have the same model structure, wherein there is a correspondence between the multiple denoising network layers corresponding to each perspective, such as the first denoising network layer corresponding to perspective A and the first denoising network layer corresponding to perspective B.

[0072] Optionally, at least one denoising network layer corresponding to multiple perspectives is a denoising network layer trained based on the perspective. Therefore, although the same network layer structure exists between multiple denoising network layers with corresponding relationships under different perspectives, there are differences in network layer parameters between the multiple denoising network layers.

[0073] Schematically, the first denoising network layer under perspective A corresponds to the second denoising network layer under perspective B, but the network layer parameter of the first denoising network layer under perspective A is represented by parameter β, and the network layer parameter of the first denoising network layer under perspective B is represented by parameter θ, where parameter β is different from parameter θ.

[0074] Among them, the input feature representation is the feature representation of the input denoising network layer.

[0075] Schematically, when the denoising network layer performs denoising based on the noisy feature representation, it also adjusts the feature dimension of the noisy feature representation. For any denoising network layer, the feature representation to be input to the denoising network layer is first determined as the input feature representation. Compared with the noisy feature representation, the input feature representation has a change in noise reduction after pre-denoising, or a change in feature dimension, etc.

[0076] In an optional embodiment, the denoising network layer includes a decoding network layer. When the noisy feature representation is denoised by the denoising network layers corresponding to multiple perspectives, the noisy feature representation is decoded by the decoding network layers corresponding to the multiple perspectives, and the data to be input into the decoding network layers corresponding to the multiple perspectives for decoding are determined as input feature representations.

[0077] Schematically, each perspective corresponds to multiple decoding network layers. For any one of the multiple perspectives, the input feature representation to be input into each decoding network layer for decoding is determined, that is, the input feature representation corresponding to the multiple decoding network layers is determined based on one perspective.

[0078] In some embodiments, each perspective corresponds to at least one denoising network layer, and there is a corresponding relationship between the at least one denoising network layer corresponding to multiple perspectives. For any denoising network layer, the input feature representations corresponding to the denoising network layer at the same level of the multiple perspectives are determined, thereby determining multiple input feature representations corresponding to the denoising network layer at the level based on the denoising network layer with corresponding relationships and multiple perspectives.

[0079] Schematically, taking the denoising network layer as the decoding network layer as an example, view 1 corresponds to two decoding network layers, and view 2 corresponds to two decoding network layers. There is a correspondence between the first decoding network layer corresponding to view 1 and the first decoding network layer corresponding to view 2; there is a correspondence between the second decoding network layer corresponding to view 1 and the second decoding network layer corresponding to view 2. Therefore, for the first decoding network layer, the input feature representations corresponding to multiple view angles (view 1 and view 2) are determined, and multiple input feature representations corresponding to the first decoding network layer are obtained; similarly, for the second decoding network layer, the input feature representations corresponding to multiple view angles (view 1 and view 2) are determined, and multiple input feature representations corresponding to the second decoding network layer are obtained, and so on.

[0080] Step 230 : extracting three-dimensional shared information shared between the three-dimensional transformation matrices corresponding to the plurality of input feature representations.

[0081] There is a corresponding relationship between at least one denoising network layer corresponding to each of the multiple perspectives; when extracting three-dimensional shared information, any one of the at least one denoising network layer is analyzed, and multiple input feature representations are determined based on the denoising network layer at the same level of the multiple perspectives.

[0082] For example: there is a corresponding relationship between the first denoising network layer corresponding to perspective A and the first denoising network layer corresponding to perspective B. The input feature representation 1 to be input to the first denoising network layer corresponding to perspective A is determined, and the input feature representation 2 to be input to the first denoising network layer corresponding to perspective B is determined. The input feature representation 1 and the input feature representation 2 are used as multiple input feature representations corresponding to the first denoising network layer.

[0083] Illustratively, the input feature representation is a feature representation obtained by denoising and / or feature scaling based on the two-dimensional noisy feature representation, and thus the input feature representation is a two-dimensional feature representation.

[0084] In some embodiments, in order to be able to study the input feature representation on a three-dimensional scale, after obtaining the input feature representations corresponding to multiple denoising network layers with corresponding relationships, the multiple input feature representations are dimensionally converted respectively, thereby obtaining three-dimensional transformation matrices corresponding to the multiple input feature representations, that is, obtaining multiple three-dimensional transformation matrices. The three-dimensional transformation matrix is ​​a three-dimensional feature representation, and there is a one-to-one correspondence between the multiple three-dimensional transformation matrices and the multiple input feature representations.

[0085] That is, the three-dimensional transformation matrix is ​​obtained by transforming the dimension of the input feature representation.

[0086] In some embodiments, after obtaining a plurality of three-dimensional transformation matrices, the plurality of three-dimensional transformation matrices are analyzed to extract three-dimensional shared information common among the plurality of three-dimensional transformation matrices.

[0087] Optionally, a pooling layer with an attention mechanism is used to analyze multiple three-dimensional transformation matrices, thereby extracting three-dimensional shared information common among the multiple three-dimensional transformation matrices.

[0088] Schematically, 3D shared information is used to represent the shared relationships between multiple input feature representations on a 3D scale. For example, 3D shared information can reveal the geometric consistency between multiple 3D transformation matrices representing multiple input feature representations, as well as various attribute information such as the shape, structure, and color of a physical element. For example, 3D shared information can be used to indicate a gap at a corner of a physical element, or a different color at the center of a physical element.

[0089] Optionally, the denoising processes performed separately at different perspectives can be constrained with the help of three-dimensional shared information, so as to avoid the denoising network layers corresponding to different perspectives performing separate denoising on the noisy feature representation only on the basis of the previous training process, thereby affecting the analytical correlation of the same noisy feature representation from multiple perspectives; that is, with the help of three-dimensional shared information representing common information, the analytical correlation between denoising network layers with corresponding relationships at multiple perspectives can be improved.

[0090] Schematically, the three-dimensional shared information is information obtained based on multiple input feature representations of multiple denoising network layers with corresponding relationships, so there is a corresponding relationship between the three-dimensional shared information and the denoising network layer; when there are multiple denoising network layers for each perspective, then from a comprehensive perspective, the three-dimensional shared information corresponding to the multiple denoising network layers under one perspective will be extracted.

[0091] For example: based on the multiple input feature representations of the first denoising network layer corresponding to multiple perspectives, three-dimensional shared information 1 is extracted, and the three-dimensional shared information 1 corresponds to the first denoising network layer; if each perspective includes multiple denoising network layers, three-dimensional shared information 2 is also extracted based on the second denoising network layer corresponding to the multiple perspectives, and the three-dimensional shared information 2 corresponds to the second denoising network layer, and so on.

[0092] In some embodiments, denoising network layers at the same level share the same or the same set of three-dimensional shared information.

[0093] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0094] Step 240 : Adjust the multiple input feature representations using the three-dimensional shared information to obtain multiple adjusted feature representations.

[0095] Schematically, three-dimensional shared information is common information extracted based on multiple input feature representations, which can comprehensively display the same content and / or similar content between the multiple input feature representations.

[0096] In some embodiments, after obtaining three-dimensional shared information corresponding to multiple input feature representations, the multiple input feature representations are adjusted respectively using the three-dimensional shared information, thereby obtaining adjusted feature representations corresponding to the multiple input feature representations, that is, obtaining multiple adjusted feature representations. That is, there is a corresponding relationship between the multiple input feature representations and the multiple adjusted feature representations. Schematically, the input feature representation 1 of the first denoising network layer under perspective A is adjusted using the three-dimensional shared information to obtain an adjusted feature representation 1' corresponding to the input feature representation 1; in addition, the input feature representation 2 of the first denoising network layer under perspective B is adjusted using the three-dimensional shared information to obtain an adjusted feature representation 2' corresponding to the input feature representation 2, and so on.

[0097] Step 250 : generating perspective images corresponding to the entity element at multiple perspectives based on the multiple adjusted feature representations.

[0098] Schematically, after obtaining multiple adjusted feature representations corresponding to multiple input feature representations, the adjusted feature representations are input into the denoising network layer corresponding to the input feature representation, thereby performing feature size conversion and / or denoising on the adjusted feature representation containing three-dimensional shared information through the denoising network layer, thereby avoiding the problem of lack of correlation when independent analysis is performed through the denoising network layers corresponding to different perspectives.

[0099] For example, taking the analysis of the first denoising network layer corresponding to multiple perspectives as an example, the input feature representation 1 of the first denoising network layer corresponding to perspective A and the input feature representation 2 of the first denoising network layer corresponding to perspective B are integrated to obtain three-dimensional shared information; the input feature representation 1 is adjusted through the three-dimensional shared information to obtain an adjusted feature representation 1' corresponding to the input feature representation 1; the input feature representation 2 is adjusted through the three-dimensional shared information to obtain an adjusted feature representation 2' corresponding to the input feature representation 2.

[0100] The adjusted feature representation 1' is input into the first denoising network layer corresponding to view A, so that the first denoising network layer corresponding to view A can process the adjusted feature representation 1' including the three-dimensional shared information; similarly, the adjusted feature representation 2' is input into the first denoising network layer corresponding to view A, so that the first denoising network layer corresponding to view B can process the adjusted feature representation 2' including the three-dimensional shared information, and so on.

[0101] Optionally, when the denoising network layer is implemented as a decoding network layer, the adjusted feature representation 1' is input to the first decoding network layer corresponding to view A, so that the first decoding network layer corresponding to view A can decode the adjusted feature representation 1' including three-dimensional shared information, such as performing feature size conversion on the adjusted feature representation 1'; similarly, the adjusted feature representation 2' is input to the first decoding network layer corresponding to view B, so that the first decoding network layer corresponding to view B can decode the adjusted feature representation 2' including three-dimensional shared information, such as performing feature size conversion on the adjusted feature representation 2' to the same scale.

[0102] In an optional embodiment, after denoising the adjusted feature representation through a denoising network layer corresponding to each perspective, a denoised feature representation corresponding to each perspective is obtained.

[0103] The denoised feature representation is a feature representation obtained by denoising the noisy feature representation.

[0104] Optionally, each perspective corresponds to multiple denoising network layers. For any perspective, the last adjusted feature representation is denoised by the last denoising network layer under the perspective, so as to obtain the denoised feature representation corresponding to the perspective.

[0105] For example: the last adjusted feature representation o is denoised by the last denoising network layer corresponding to view A, thereby obtaining the denoised feature representation o' corresponding to view A; similarly, the last adjusted feature representation p is denoised by the last denoising network layer corresponding to view B, thereby obtaining the denoised feature representation p' corresponding to view B, and so on.

[0106] In an optional embodiment, a plurality of denoising feature representations are processed separately by a decoder to obtain perspective images corresponding to the plurality of denoising feature representations, thereby obtaining a plurality of perspective images.

[0107] In schematic form, a decoder is used to decode multiple denoised feature representations respectively, so as to restore the perspective image based on the denoised feature representation; the multiple denoised feature representations correspond one-to-one to the multiple perspectives, and the restored perspective image is the predicted image content when the entity element is observed through the corresponding perspective.

[0108] For example: perspective A corresponds to the denoised feature representation o', and the denoised feature representation o' is decoded by the decoder to restore the perspective image a corresponding to perspective A. The perspective image a is the predicted image content when the entity element is observed through the corresponding perspective A; similarly, perspective B corresponds to the denoised feature representation p', and the denoised feature representation p' is decoded by the decoder to restore the perspective image b corresponding to perspective B. The perspective image b is the predicted image content when the entity element is observed through the corresponding perspective B, and so on.

[0109] Optionally, multiple perspectives correspond to one encoder respectively, and the denoising feature representation corresponding to the perspective is encoded by the encoder corresponding to the perspective; or, multiple perspectives correspond to one encoder, and the denoising feature representation corresponding to each perspective is encoded by the encoder, which is not limited here.

[0110] Among them, multiple perspective images are used to integrate and generate a three-dimensional model representing the entity elements.

[0111] Schematically, multiple perspective images are image contents that display entity elements from different perspectives. After obtaining the multiple perspective images, the multiple perspective images are processed with the help of a preset program, so that the multiple perspective images are integrated to generate a three-dimensional model representing the entity elements. The three-dimensional model can display the entity elements from multiple perspectives, and the shape, structure and other attribute information of the three-dimensional model conforms to the multiple perspective images.

[0112] In some embodiments, in order to make the three-dimensional model closer to the real physical elements, more perspectives are selected to perform the above process, such as: selecting three or more perspectives to perform the above denoising process, so that the physical elements can be described more carefully and accurately, and a more realistic three-dimensional model can be obtained.

[0113] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0114] In summary, by comprehensively analyzing multiple input feature representations, the same information of multiple input feature representations on a three-dimensional scale can be obtained, thereby extracting three-dimensional shared information. The input feature representation is constrained with the help of three-dimensional shared information, thereby avoiding the problem of separation when different perspectives are independently denoised through the corresponding denoising network layer. Adjusting the input feature representation through three-dimensional shared information is conducive to strengthening the correlation in the denoising process of different perspectives, thereby making the perspective images have a stronger correlation, which is conducive to generating a three-dimensional model with stronger geometric consistency based on the integration of multiple perspective images, and improving the authenticity and meticulousness of the entity elements represented by the three-dimensional model.

[0115] In an optional embodiment, when extracting 3D shared information, the 3D transformation matrices corresponding to the multiple input feature representations are first obtained. Then, a pooling layer with an attention mechanism is used to analyze the multiple 3D transformation matrices to extract the 3D shared information representing the shared structural information. As shown in FIG3 , step 230 of FIG2 can also be implemented as steps 310 to 320 as follows.

[0116] In step 310 , the plurality of input feature representations are back-projected to obtain three-dimensional transformation matrices corresponding to the plurality of input feature representations.

[0117] Schematically, the input feature representation is a feature representation of a two-dimensional scale, for example, the input feature representation is a feature represented in the form of a two-dimensional matrix.

[0118] Optionally, after obtaining the multiple input feature representations, in order to adjust the two-dimensional input feature representations to a three-dimensional scale representing spatial information, the multiple input feature representations are back-projected respectively.

[0119] Among them, the back projection processing refers to reconstructing a three-dimensional object space based on a two-dimensional image, mapping the input feature representation to the three-dimensional object space based on the object space, and the back projection processing representation performed on the input feature representation reconstructs the two-dimensional input feature representation to obtain a three-dimensional transformation matrix.

[0120] In an optional embodiment, multiple input feature representations are back-projected to obtain projection feature representations corresponding to multiple viewing angles.

[0121] Optionally, the input feature representation is back-projected using a pre-selected back-projection function.

[0122] Schematically, multiple input feature representations are back-projected using a back-projection function, thereby converting each input feature representation into a three-dimensional projected feature representation. In other words, the projected feature representation is used to represent the feature information of the input feature representation in three dimensions, and also represents the input feature representation in spatial dimensions.

[0123] Among them, when processing multiple denoising network layers with corresponding relationships, the multiple input feature representations obtained correspond one-to-one to multiple perspectives. Therefore, after back-projecting the input feature representations respectively, the multiple projection feature representations obtained correspond one-to-one to multiple perspectives.

[0124] Optionally, in addition to converting the input feature representation into a three-dimensional projection feature representation, in order to more highlight the impact of different perspectives on the projection feature representation, parameter feature representations corresponding to multiple perspectives are also obtained. The parameter feature representation is a feature representation obtained based on the camera parameters corresponding to the perspective, and is used to characterize the spatial information under the perspective.

[0125] There is a corresponding relationship between multiple parameter feature representations and multiple projection feature representations.

[0126] In some embodiments, camera parameters corresponding to multiple perspectives are obtained.

[0127] Schematically, the viewing angle corresponds to the camera, and the determination of the viewing angle is affected not only by the position of the camera, but also by the shooting direction of the camera. The camera is an abstract expression used to represent the acquisition of physical elements. Based on the selection of multiple viewing angles, the camera parameters represented by the cameras corresponding to the multiple viewing angles can be determined.

[0128] Based on the correspondence between multiple perspectives and camera parameters, the relative position between the camera and the expected physical elements is represented by the camera parameters.

[0129] Optionally, the camera parameters include camera position and camera direction, where the camera position represents the position of the camera, and the camera direction represents the direction of the camera; for example, the camera position represents the position of the camera relative to the physical element in the world coordinate system, and the camera direction represents the shooting direction of the camera relative to the physical element in the world coordinate system.

[0130] It is worth noting that the camera expressed here is an abstract expression of the perspective, and the relative position between the camera and the physical element is called the camera parameters. The camera parameters are predetermined known parameters, but the perspective image obtained based on the camera / perspective is not obtained in advance, but is the image content predicted based on information such as the camera parameters. That is, the purpose of determining the camera parameters is to perform a predictive analysis process through the corresponding perspective, so as to generate a perspective image of the corresponding perspective based on the perspective corresponding to the camera position.

[0131] In some embodiments, volume expressions are performed on the plurality of camera parameters respectively to obtain parameter volume expressions corresponding to the plurality of camera parameters respectively.

[0132] Schematically, the parametric volume representation is a feature representation obtained by expressing camera parameters in three-dimensional space.

[0133] By obtaining the perspective acquisition rules of the perspective images corresponding to the perspective images generated by the camera parameter expression, the camera parameters are first converted into three-dimensional space expressions to obtain parameter volume expressions based on the perspective acquisition rules corresponding to the camera parameters. The parameter volume expressions are then encoded based on the encoding function to obtain parameter feature representation. On this basis, the three-dimensional transformation matrix of the input feature representation is obtained, thereby improving the generation efficiency and accuracy of the three-dimensional transformation matrix.

[0134] Optionally, in the three-dimensional space in which the back-projection process is performed on the input feature representation, a plurality of voxels in the three-dimensional space are determined.

[0135] Voxel is the abbreviation of volume pixel, which is a combination of pixel, volume and element, equivalent to a pixel in three-dimensional (3D) space.

[0136] Illustratively, the plurality of voxels are implemented as all voxels constituting the three-dimensional space; or, the plurality of voxels are implemented as part of the voxels in the three-dimensional space.

[0137] Optionally, the three-dimensional space projected by the back-projection processing is a space composed of a large number of voxels. When volume expressions are performed on multiple camera parameters respectively, the camera corresponding to each camera parameter and the three-dimensional space projected during the back-projection processing are determined, and the camera and multiple voxels in the three-dimensional space are integrated to obtain a parameter volume expression corresponding to the camera parameter.

[0138] Schematically, taking the analysis of camera parameter Q among multiple camera parameters as an example, the camera corresponding to the camera parameter Q is camera C, the corresponding viewing angle is viewing angle A, viewing angle A corresponds to input feature representation 1, and the three-dimensional space projected by input feature representation 1 is three-dimensional space S. Multiple voxels in the three-dimensional space S (such as voxel s1, voxel s2, etc.) are determined, so as to obtain the parameter volume expression corresponding to the camera parameter Q by integrating camera C and multiple voxels in the three-dimensional space S.

[0139] Optionally, the relative position between the camera and multiple voxels is determined to obtain the viewing direction and viewing depth corresponding to the multiple voxels respectively; the viewing direction and viewing depth corresponding to each voxel are recorded in the voxel, and the multiple voxels are integrated to obtain a parameter volume expression corresponding to the camera parameters.

[0140] The viewing direction is determined based on the direction of the voxel relative to the camera center; the viewing depth is determined based on the distance of the voxel relative to the camera center.

[0141] Schematically, each voxel is connected to the camera center of the corresponding camera, and the direction of the connection line is recorded as the viewing direction; the projection length of the connection line in the direction of the camera's main optical axis is recorded as the viewing depth.

[0142] For example, let's analyze camera parameter Q among multiple camera parameters. The camera corresponding to camera parameter Q is camera C, and the corresponding three-dimensional space is three-dimensional space S, which includes multiple voxels such as voxel s1 and voxel s2. For voxel s1, the viewing direction of voxel s1 relative to the camera center c of camera C is determined, as is the viewing depth of voxel s1 relative to the camera center c of camera C. Furthermore, the viewing direction of voxel s2 relative to the camera center c of camera C is determined, as is the viewing depth of voxel s2 relative to the camera center c of camera C. The viewing directions and depths recorded by multiple voxels are then combined to obtain a parametric volume expression corresponding to the camera parameter.

[0143] Optionally, the parameter volume expressions corresponding to the plurality of camera parameters are determined respectively through the above process.

[0144] In some embodiments, the parameter volume expression is encoded by a preset feature encoding function to obtain parameter feature representations corresponding to multiple viewing angles.

[0145] Among them, the preset feature encoding function is used to normalize the parameter volume expression to obtain multiple parameter feature representations for the parameter volume expressions corresponding to multiple perspectives, and the multiple parameter feature representations correspond one-to-one to the multiple parameter volume expressions.

[0146] In an optional embodiment, based on the corresponding relationship, the projection feature representation and the parameter feature representation under the same viewing angle are spliced ​​to obtain a three-dimensional transformation matrix corresponding to the multiple input feature representations.

[0147] Schematically, after obtaining multiple parameter feature representations and multiple projection feature representations, since the multiple parameter feature representations correspond one-to-one to multiple viewpoints, the multiple projection feature representations also correspond one-to-one to the multiple viewpoints, thereby determining a one-to-one correspondence between the multiple parameter feature representations and the multiple projection feature representations; based on the correspondence, the projection feature representation and the parameter feature representation corresponding to the same viewpoint are feature splicing processed to obtain a three-dimensional transformation matrix corresponding to each viewpoint, that is, a three-dimensional transformation matrix corresponding to multiple input feature representations is obtained.

[0148] In some embodiments, the position point represented by each perspective is called a viewpoint, and the task of generating a three-dimensional model through N viewpoints is taken as an example for explanation.

[0149] Schematically, for a generation task of N viewpoints, the viewpoints are numbered i=1,…,N.

[0150] Optionally, taking the denoising network layer implemented as a decoding network layer as an example, the input is N feature images (N input feature representations).

[0151] Schematically, the input feature representation of viewpoint i is recorded as m i , for m i Perform back projection to obtain the three-dimensional transformation matrix v back projected into the three-dimensional space i , as shown in the following formula 1.

[0152] Formula 1:

[0153] Among them, -1 A back-projection function that converts the two-dimensional input feature representation into a three-dimensional projection feature representation; Indicates the feature concatenation processing of feature dimensions; PosEncode indicates the preset feature encoding function; Represents a parametric volume representation of the camera parameters, including the viewing direction and depth.

[0154] In step 320 , attention pooling is performed on the multiple three-dimensional transformation matrices to obtain a volume feature representation.

[0155] Optionally, an attention pooling process is performed through a pooling layer with an attention mechanism; after obtaining multiple three-dimensional transformation matrices for a denoising network layer with a corresponding relationship, in order to analyze the correlated spatial information represented by the multiple three-dimensional transformation matrices, attention pooling processing is performed on the multiple three-dimensional transformation matrices to focus on the three-dimensional shared information shared between the multiple three-dimensional transformation matrices.

[0156] Volumetric feature representation is used to characterize the three-dimensional shared information between multiple three-dimensional transformation matrices.

[0157] In an optional embodiment, a plurality of voxel sets represented by respective three-dimensional transformation matrices are determined.

[0158] The three-dimensional transformation matrix is ​​a matrix obtained by comprehensively representing the projection feature representation and the parameter feature representation. Therefore, multiple three-dimensional transformation matrices can respectively represent the feature information of the corresponding perspective.

[0159] Optionally, when determining the voxel sets represented by the multiple three-dimensional transformation matrices, the three-dimensional space in which the three-dimensional transformation matrix is ​​obtained is determined, and the multiple voxels in the three-dimensional space that participate in determining the parametric volume expression are grouped into the voxel set. For example, all voxels in the three-dimensional space are grouped into the voxel set.

[0160] Schematically, the voxel set is used to represent a set of multiple voxels in a three-dimensional space when obtaining a three-dimensional transformation matrix.

[0161] In an optional embodiment, multiple attention values ​​are determined based on multiple voxels at the same voxel position in multiple voxel sets. Schematically, multiple voxels at the same voxel position in multiple voxel sets are focused on to obtain multiple attention values.

[0162] Optionally, each 3D transformation matrix corresponds to a voxel set, resulting in multiple voxel sets. Because the position encoding of multiple voxels in the 3D space represented by the parametric volume is performed when the 3D transformation matrix is ​​obtained, the positions of identical voxels in the multiple voxel sets can be quickly determined based on the position encoding process.

[0163] Schematically, taking any voxel position as an example, the voxel corresponding to the voxel position in each voxel set is determined to obtain multiple voxels; multiple voxels are paid attention to in parallel to extract the common information between different voxels belonging to the same voxel position, and the common information is represented by the attention value.

[0164] Similarly, the above analysis is performed on multiple voxel positions respectively, so that multiple voxels corresponding to the same voxel position are focused on in parallel to extract the common information corresponding to the multiple voxel positions, thereby obtaining the attention values ​​corresponding to the multiple voxel positions.

[0165] In an optional embodiment, multiple attention values ​​are pooled to obtain a volume feature representation.

[0166] Schematically, multiple attention values ​​are pooled together to obtain a volume feature representation that combines multiple shared information. This volume feature representation not only performs parallel analysis of voxels corresponding to the same voxel position under multiple perspectives, but also performs a comprehensive analysis based on the attention values ​​corresponding to different voxel positions. It can more accurately and meticulously display the three-dimensional shared information shared by multiple input feature representations.

[0167] In some embodiments, the three-dimensional transformation matrix v calculated by the above formula 1 is i , through the attention pooling mechanism, the three-dimensional transformation matrix v of N viewpoints is transformed i Unified into a volume feature representation As shown in the following formula 2.

[0168] Formula 2:

[0169] Among them, Pool is used to represent pooling processing; It is used to represent the Hadamard product, which is a type of matrix calculation used to represent the one-to-one multiplication between matrix elements; Attention is used to represent the attention layer for attention processing; x is used to represent the voxel (or voxel block) in the three-dimensional transformation matrix; v i [x] is used to represent the relationship between voxels corresponding to the same voxel position at multiple different perspectives, which can be regarded as the above-mentioned attention value; Attention({v i [x]i=1…N}) is used to coordinate attention values ​​corresponding to multiple voxel positions.

[0170] In summary, by comprehensively analyzing multiple input feature representations, the same information of multiple input feature representations on a three-dimensional scale can be obtained, thereby extracting three-dimensional shared information. The input feature representation is constrained with the help of three-dimensional shared information, thereby avoiding the problem of fragmentation when different perspectives are independently denoised through the corresponding denoising network layer. Adjusting the input feature representation through three-dimensional shared information is conducive to strengthening the correlation during the denoising process of different perspectives, thereby making the perspective images have a stronger correlation, which is conducive to generating a three-dimensional model with stronger geometric consistency based on the integration of multiple perspective images, and improving the authenticity and meticulousness of the entity elements represented by the three-dimensional model.

[0171] In the embodiment of the present application, the content of extracting three-dimensional shared information is introduced. First, the input feature representation of the two-dimensional scale is back-projected to obtain a three-dimensional transformation matrix corresponding to the input feature representation; then the three-dimensional transformation matrix is ​​subjected to attention pooling processing to obtain a volume feature representation as a feature expression form of the three-dimensional shared information. By converting the two-dimensional scale into a three-dimensional scale, it is beneficial to perform a more targeted analysis of the geometric information represented by the input feature representation, so that the three-dimensional shared information can be more finely represented with the help of the volume feature representation, which is beneficial to the analysis in the spatial dimension based on the volume feature representation, and is also beneficial to the subsequent adjustment process based on the volume feature representation.

[0172] In an optional embodiment, when adjusting the input feature representation using three-dimensional shared information, a dimensionality conversion is first performed based on the three-dimensional shared information to obtain a two-dimensional residual feature representation. This residual feature representation is then concatenated with the input feature representation to obtain an adjusted feature representation. As shown in FIG4 , step 240 of FIG2 can be implemented as steps 410 to 430 as follows.

[0173] Step 410: Obtain a volume feature representation representing the three-dimensional shared information.

[0174] Schematically, multiple input feature representations are respectively dimensionally transformed to obtain a three-dimensional transformation matrix. The three-dimensional shared information obtained through the multiple three-dimensional transformation matrices is information represented in a three-dimensional scale, and the three-dimensional shared information of the three-dimensional scale can be represented by a volume feature representation.

[0175] Schematically, the volume feature representation is a feature representation determined by a world coordinate system, which is a coordinate system established based on a selected origin, horizontal axis, vertical axis, and vertical axis.

[0176] Step 420 : Based on the perspectives and volume feature representations corresponding to the multiple input feature representations, obtain three-dimensional feature representations corresponding to the multiple perspectives.

[0177] Schematically, the volume feature representation is a feature representation that corresponds to multiple input feature representations; when adjusting the input feature representation through the volume feature representation, the perspectives corresponding to the multiple input feature representations are taken into consideration, so as to combine the perspective and the volume feature representation to obtain a three-dimensional feature representation containing perspective information.

[0178] Among them, the three-dimensional feature representation is used to characterize the impact of the volume feature representation on the input feature representation in the spatial dimension; it can also indirectly show the role of viewing angle when the volume feature representation adjusts the input feature representation.

[0179] Based on the one-to-one correspondence between multiple three-dimensional feature representations and multiple perspectives, there is a one-to-one correspondence between multiple input feature representations and multiple perspectives, that is, there is a correspondence between multiple three-dimensional feature representations and multiple input feature representations.

[0180] In an optional embodiment, camera coordinate systems corresponding to multiple perspectives are determined.

[0181] The camera coordinate system is a coordinate system established based on the camera used when determining the corresponding viewing angle as a reference point. The camera coordinate system is a coordinate system established based on the camera used when determining the viewing angle as a reference point.

[0182] Schematically, multiple perspectives are position information determined based on the camera position and camera orientation of different cameras, so each perspective corresponds to a camera. The camera coordinate system is a coordinate system established by determining the origin, horizontal axis, vertical axis, and vertical axis with the camera as the reference point. For example: the camera center is used as the origin of the camera coordinate system, and the horizontal, vertical, and vertical axes are selected to create the camera coordinate system; or, any point on the camera is used as the origin, and the horizontal, vertical, and vertical axes are selected to create the camera coordinate system, etc.

[0183] Optionally, a camera coordinate system is established based on each camera, corresponding to each of the multiple viewpoints. Then, multiple camera coordinate systems are created through the multiple cameras, and each of the multiple viewpoints corresponds to a camera coordinate system.

[0184] In an optional embodiment, the volume feature representation is three-dimensionally mapped with reference to the camera coordinate system corresponding to the viewing direction, and the volume feature representation is mapped to a three-dimensional space to obtain coordinate feature representations corresponding to multiple viewing angles.

[0185] Optionally, the volume feature representation is a three-dimensional scale feature representation displayed through a three-dimensional space, where the three-dimensional space is established based on a world coordinate system, and the world coordinate system can map different feature representations to the same vector space.

[0186] Schematically, the volume feature representation is three-dimensionally mapped with reference to the camera coordinate system corresponding to each perspective, so that the volume feature representation displayed in the world coordinate system is mapped to the camera coordinate system corresponding to each perspective, and the coordinate feature representation corresponding to multiple perspectives is obtained.

[0187] The coordinate feature representations corresponding to different perspectives are feature representations obtained based on different cameras as reference points. Therefore, there may be certain differences between the coordinate feature representations corresponding to multiple perspectives.

[0188] For example: perspective A corresponds to camera coordinate system a. With camera coordinate system a as a reference, the volume feature representation in the world coordinate system is mapped to camera coordinate system a, thereby realizing the three-dimensional coordinate mapping process of the volume feature representation, and obtaining the coordinate feature representation 1 corresponding to perspective A; similarly, perspective B corresponds to camera coordinate system b. With camera coordinate system b as a reference, the volume feature representation in the world coordinate system is mapped to camera coordinate system b, thereby realizing the three-dimensional coordinate mapping process of the volume feature representation, and obtaining the coordinate feature representation 2 corresponding to perspective B, and so on.

[0189] In some embodiments, the coordinate conversion process is performed using a preset three-dimensional coordinate conversion function. Schematically, the volume feature representation in the world coordinate system is converted into a coordinate feature representation in the camera coordinate system through trilinear interpolation.

[0190] In an optional embodiment, based on the coordinate feature representations corresponding to the multiple perspectives, three-dimensional feature representations corresponding to the multiple perspectives in the three-dimensional space are obtained.

[0191] Illustratively, after obtaining coordinate feature representations corresponding to multiple viewing angles, at the same viewing angle, a three-dimensional feature representation of the viewing angle in three-dimensional space is obtained based on the viewing angle and the coordinate feature representation.

[0192] In some embodiments, perspective depths represented by multiple perspectives are obtained.

[0193] Schematically, the depth of view is determined based on the distance of the voxel relative to the camera center, where the voxel is a component element in the three-dimensional space. Each voxel is connected to the camera center of the corresponding camera, and the projection length of the connecting line in the direction of the camera's main optical axis is recorded as the depth of view, so that each voxel corresponds to a depth of view.

[0194] Optionally, the multiple perspectives correspond to multiple voxels respectively, and each voxel corresponds to a perspective depth.

[0195] In some embodiments, at the same viewing angle, three-dimensional feature representations corresponding to multiple viewing angles in three-dimensional space are obtained based on the viewing angle depth and coordinate feature representation.

[0196] After obtaining the viewing depth, the distance between each voxel and the camera center is represented by the viewing depth, so that the voxel is expressed in three dimensions, which improves the accuracy of the three-dimensional conversion of the viewing direction.

[0197] Schematically, analysis is performed on any viewing angle to determine the coordinate feature representation corresponding to the viewing angle and the viewing depths corresponding to multiple voxels at the viewing angle.

[0198] Optionally, a set of voxels represented by three-dimensional transformation matrices corresponding to a plurality of viewing angles is determined.

[0199] Schematically, the voxel set includes multiple voxels, and each voxel corresponds to a viewing depth.

[0200] Optionally, the viewing angle depth corresponding to the voxel is used as the voxel value, and multiple voxel blocks in the voxel set are filled respectively to obtain a voxel block set with voxel values.

[0201] Schematically, taking the voxel set S corresponding to perspective A as an example, the voxel set S corresponding to perspective A includes voxel s1, voxel s2, etc., the perspective depth corresponding to voxel s1 is the voxel value corresponding to voxel s1, the perspective depth corresponding to voxel s2 is the voxel value corresponding to voxel s2, and so on; based on the above-mentioned filling process of the voxel set, a plurality of voxel blocks with voxel values ​​are obtained, that is, a voxel block set with voxel values ​​is obtained.

[0202] In some embodiments, based on the voxel block set and the coordinate feature representation, three-dimensional feature representations corresponding to multiple perspectives in the three-dimensional space are obtained.

[0203] Illustratively, multiple viewpoints each correspond to a voxel block set, resulting in multiple voxel block sets; multiple viewpoints each correspond to a coordinate feature representation, resulting in multiple coordinate feature representations; based on the voxel block set and coordinate feature representation at the same viewpoint, a 3D feature representation corresponding to that viewpoint is obtained, resulting in multiple 3D feature representations. Filling the voxels in the voxel set with their viewpoint depth as the voxel value improves the voxel block set's ability to express 3D information, and thus improves the 3D feature representation's ability to express 3D information.

[0204] In some embodiments, the voxel block set is position-encoded using a preset encoding function to obtain a voxel feature representation. The voxel block set is encoded using a preset encoding function to obtain a voxel feature representation.

[0205] Schematically, a position encoding process is performed on a set of voxel blocks having voxel values ​​using a preset encoding function, and the result after the position encoding is called a voxel feature representation.

[0206] In some embodiments, at the same viewing angle, the voxel feature representation and the coordinate feature representation are spliced ​​to obtain three-dimensional feature representations corresponding to multiple viewing angles in three-dimensional space.

[0207] Schematically, at the same viewing angle, the voxel feature representation and the coordinate feature representation are feature spliced ​​to obtain a three-dimensional feature representation at that viewing angle; based on the feature splicing processing of the voxel feature representation and the coordinate feature representation corresponding to multiple viewing angles, three-dimensional feature representations corresponding to multiple viewing angles are obtained, that is, multiple three-dimensional feature representations are obtained.

[0208] In some embodiments, the volume feature calculated by the above formula 2 is represented as First, the volume feature is represented by coordinate mapping Processing is performed to express the three-dimensional features in the camera coordinate system corresponding to each viewpoint using the difference As shown in the following formula 3:

[0209] Formula 3:

[0210] Among them, Warp is a three-dimensional coordinate conversion function used to represent the volume features in the world coordinate system Convert to the volume expression in the camera coordinate system (i.e. the coordinate feature representation mentioned above); It is a volume feature representation abbreviation of; Indicates the feature concatenation processing of feature dimensions; PosEncode is a preset encoding function; Represents a voxel feature representation determined based on the camera depth in the camera parameters.

[0211] Based on the camera coordinate system corresponding to each viewing angle, the coordinate feature representations corresponding to multiple viewing angles are determined. After conversion to three-dimensional space, the three-dimensional feature representations corresponding to each viewing angle are obtained, thereby improving the efficiency and accuracy of three-dimensional conversion of each viewing angle.

[0212] Step 430 : Based on the correspondence, a plurality of adjusted feature representations are obtained using the three-dimensional feature representation and the input feature representation at the same viewing angle.

[0213] Schematically, after determining that the corresponding three-dimensional feature representation is obtained with the help of the camera coordinate system corresponding to multiple perspectives, multiple three-dimensional feature representations are obtained. Based on the correspondence between the multiple three-dimensional feature representations and the multiple perspectives, and the correspondence between the multiple input feature representations and the multiple perspectives, the adjusted feature representation is obtained through the three-dimensional feature representation and the input feature representation under the same perspective.

[0214] Optionally, considering that the three-dimensional feature representation is a feature representation of a three-dimensional scale and the input feature representation is a feature representation of a two-dimensional scale, if the three-dimensional feature representation and the input feature representation need to be analyzed together, the three-dimensional feature representation needs to be restored to a feature representation of a two-dimensional scale.

[0215] In some embodiments, the three-dimensional feature representations corresponding to the multiple perspectives are projected into a two-dimensional space to obtain residual feature representations corresponding to the multiple perspectives.

[0216] Illustratively, contrary to the above-mentioned back-projection process, a projection process is used to project the three-dimensional feature representation to obtain a residual feature representation on a two-dimensional scale.

[0217] Optionally, to facilitate the subsequent feature processing process, the projection dimension used in the back-projection process is set to be the same as the projection dimension used in the projection process. The three-dimensional feature representations corresponding to the multiple viewpoints are projected with the same projection dimension, so that the feature representation projected to the two-dimensional space is called the residual feature representation, that is, the residual feature representation corresponding to the multiple viewpoints is obtained. The three-dimensional feature representations corresponding to the multiple viewpoints are projected to the two-dimensional space, and then spliced ​​with the input feature representation to obtain the adjusted feature representation, so that the residual feature representation projected to the two-dimensional space and the input feature representation are feature representations of the same dimension, thereby improving the convenience of feature splicing and subsequent calculation processing.

[0218] In some embodiments, when projecting the three-dimensional feature representation using projection, the following formula 4 is used to project each three-dimensional feature representation into a residual feature representation through an attention pooling operation of the same projection dimension:

[0219] Formula 4:

[0220] In the back-projection process, a pixel of a two-dimensional scale will project a ray, and pixels of multiple two-dimensional scales will project multiple rays to form a three-dimensional space; accordingly, in the projection process, each ray will be projected into a pixel of a corresponding two-dimensional scale.

[0221] in, Used to represent three-dimensional features with the same projection dimension Perform projection processing; r is used to represent the pixel (i.e., ray in three-dimensional space) in the residual feature representation of two-dimensional scale (which can be regarded as an image feature); d is used to represent the depth on the ray. In the projection restoration process, for d from the plane d near To the far plane d far Uniform sampling in .

[0222] Optionally, Through a multilayer perceptron (MLP), it is transformed into the above input feature representation m i Residual feature representation with the same size.

[0223] In some embodiments, at the same viewing angle, the residual feature representation and the input feature representation are concatenated to obtain adjusted feature representations corresponding to multiple viewing angles.

[0224] Schematically, residual feature representations corresponding to multiple perspectives are obtained, and based on the correspondence between the multiple residual feature representations and the multiple input feature representations determined based on the perspective, at the same perspective, feature splicing processing is performed on the residual feature representation corresponding to the perspective and the input feature representation corresponding to the perspective to obtain adjusted feature representations corresponding to the perspective; based on performing the above process on multiple perspectives respectively, adjusted feature representations corresponding to multiple perspectives are obtained.

[0225] Optionally, feature concatenation is performed on the residual feature representation and the input feature representation in the same feature dimension to obtain an adjusted feature representation.

[0226] In summary, in the embodiments of the present application, the content of adjusting the input feature representation through three-dimensional shared information is introduced. The perspective and volume feature representations are combined to obtain a three-dimensional feature representation, and then, under the same perspective, the three-dimensional feature representation is projected into a two-dimensional space to obtain residual feature representations corresponding to multiple perspectives. The input feature representation is adjusted with the help of the residual feature representation, so that the adjusted feature representation contains the three-dimensional shared information represented by the residual feature representation and also corresponds to the perspective, thereby facilitating a more accurate denoising process based on the adjusted feature representation, so as to obtain more detailed multi-perspective images with correlation between multiple perspectives.

[0227] In an optional embodiment, during the denoising process of the noisy feature representation, image generation data is obtained as a guiding condition for the denoising process. Schematically, as shown in FIG5 , the embodiment shown in FIG2 can also be implemented as the following steps 510 to 560 .

[0228] Step 510: Obtain a noise-added feature representation corresponding to the noise data.

[0229] The noise-added feature representation is used to perform denoising processing under multiple viewing angles to obtain the viewing angle images corresponding to the entity elements under the multiple viewing angles.

[0230] Schematically, a solid element is a three-dimensional solid element, which is an element that needs to be synthesized into a three-dimensional model; multiple perspectives are different perspectives, and a perspective is used to represent the observation angle adopted when observing the solid element.

[0231] Step 520: Acquire image generation data.

[0232] The image generation data is the data collected from the entity elements.

[0233] Illustratively, in the process of performing denoising based on the noisy feature representation, it is necessary to set a guiding condition to perform a targeted denoising process on the noisy feature representation.

[0234] Illustratively, the image generation data is used to generate a perspective image representing the entity element. Optionally, the image generation data is used to depict the entity element.

[0235] In an optional embodiment, at least one image data collected for the entity element is obtained as image generation data.

[0236] The image data is an image captured from a preset viewing angle for a physical element.

[0237] Schematically, the preset perspective is a pre-set perspective, which is implemented as at least one perspective, such as perspective A, perspective K, etc.; an image capture process is performed on the entity element from the preset perspective, thereby capturing at least one image data.

[0238] Optionally, when capturing images, image data is captured by a camera corresponding to a preset viewing angle.

[0239] Schematically, image data 1 is acquired by capturing the camera corresponding to the preset viewing angle A, and image data 1 is used as image generation data; alternatively, image data 1 is acquired by capturing the camera corresponding to the preset viewing angle A; image data 2 is acquired by capturing the camera corresponding to the preset viewing angle B, and image data 1 and image data 2 are used as image generation data; alternatively, image data 1 and image data 2 are acquired by capturing the camera corresponding to the preset viewing angle A, and image data 1 and image data 2 are used as image generation data, etc.

[0240] In some embodiments, the multiple perspectives used to analyze the noise-added feature representation may include the above-mentioned preset perspectives or may not include the above-mentioned preset perspectives, and this embodiment of the present application is not limited to this.

[0241] In an optional embodiment, text data for describing entity elements is obtained as image generation data.

[0242] Illustratively, text data is data obtained by describing an entity element and is used to present the entity element in text form. For example, text data may be a rabbit eating a carrot, or text data may be a virtual warrior standing in a virtual scene.

[0243] Step 530 , when denoising the noisy feature representations through the denoising network layers corresponding to the multiple perspectives, the image generation data is used as the denoising condition to determine the input feature representations of the denoising network layers corresponding to the multiple perspectives.

[0244] Schematically, the image generation data is used as a denoising condition, that is, the adjustment method of guiding the noise-added feature representation during the denoising process.

[0245] Optionally, the image generation data obtained by collecting the entity elements is obtained as a guiding condition used in the denoising process, so that in the process of denoising the noisy feature representation, the image generation data is used as reference information and the denoising process is performed towards the information represented by the image generation data.

[0246] Illustratively, the denoising condition is used to determine the noise prediction when performing denoising on a noisy feature representation. When denoising the noisy feature representation, the noise data for denoising (denoising) the noisy feature representation is predicted with the goal of reducing the noise in the noisy feature representation, using the image generation data as reference information.

[0247] Optionally, the noise data predicted based on the image generation data can be used to obtain a feature representation after removing the noise data during the process of denoising the noisy feature representation through the denoising network layer; if there are multiple denoising network layers, the corresponding noise data will be obtained based on the multiple denoising network layers respectively, and the denoising process will be performed based on the denoised data corresponding to each denoising network layer.

[0248] In some embodiments, multiple perspectives correspond to at least one denoising network layer respectively, and there is a corresponding relationship between at least one denoising network layer under multiple perspectives; then, the denoising network layer with the corresponding relationship is used to determine the input feature representation of the denoising network layer corresponding to the multiple perspectives, and the input feature representation is the feature representation to be input into the denoising network layer for denoising.

[0249] Schematically, the denoising network layer is implemented as a decoding network layer. When the noisy feature representation is decoded by the decoding network layers corresponding to multiple perspectives, the image generation data is used as the denoising condition to determine the input feature representation of the decoding network layers corresponding to the multiple perspectives.

[0250] In an optional embodiment, the multiple perspectives correspond to m denoising network layers respectively, where m is a positive integer.

[0251] Schematically, m is the number of network layers of the denoising network layers corresponding to multiple perspectives, and there is a corresponding relationship between the m denoising network layers corresponding to multiple perspectives.

[0252] For example: the first denoising network layer under perspective A corresponds to the first denoising network layer under perspective B; the mth denoising network layer under perspective A corresponds to the mth denoising network layer under perspective B, etc.

[0253] Schematically, as shown in Figure 6, two perspectives among multiple perspectives are used as an example for explanation. Each perspective corresponds to at least one denoising autoencoder (to facilitate the presentation of the iterative denoising process, it is shown that each perspective corresponds to a string of denoising autoencoders), such as perspective A corresponds to denoising autoencoder 610, perspective B corresponds to denoising autoencoder 620, etc. The denoising autoencoder is used to perform the denoising process.

[0254] Each denoising autoencoder includes multiple denoising network layers, which are used to perform the denoising process and the feature size conversion process; the above analysis is performed using the decoding network layer in the denoising network layer as an example.

[0255] Multiple denoising network layers correspond to each other. As a component of the denoising network layer, the decoding network layer also corresponds to each other. As shown in Figure 6, the decoding network layer 611 corresponding to view A corresponds to the decoding network layer 621 corresponding to view B, and the decoding network layer 612 corresponding to view A corresponds to the decoding network layer 622 corresponding to view B, and so on.

[0256] Step 540 : extracting three-dimensional shared information shared between the three-dimensional transformation matrices corresponding to the plurality of input feature representations.

[0257] Schematically, each perspective corresponds to at least one denoising network layer, and there is a one-to-one correspondence between at least one denoising network layer corresponding to multiple perspectives; when extracting three-dimensional shared information, any one of the at least one denoising network layers is analyzed, and based on the denoising network layer under multiple perspectives, multiple input feature representations corresponding to the decoding network layer are determined.

[0258] Illustratively, the input feature representation is a feature representation obtained by performing denoising processing and / or feature scale conversion on the two-dimensional noisy feature representation, and thus the input feature representation is a two-dimensional feature representation.

[0259] In some embodiments, dimension conversion is performed on multiple input feature representations respectively, thereby obtaining three-dimensional transformation matrices corresponding to the multiple input feature representations respectively, that is, the three-dimensional transformation matrix is ​​obtained by dimension conversion of the input feature representation.

[0260] In some embodiments, after obtaining a plurality of three-dimensional transformation matrices, the plurality of three-dimensional transformation matrices are analyzed to extract three-dimensional shared information common among the plurality of three-dimensional transformation matrices.

[0261] Optionally, a pooling layer with an attention mechanism is used to analyze multiple three-dimensional transformation matrices, thereby extracting three-dimensional shared information common among the multiple three-dimensional transformation matrices.

[0262] Schematically, the three-dimensional shared information is used to characterize the shared association relationship between multiple input feature representations on a three-dimensional scale.

[0263] In an optional embodiment, each of the multiple perspectives corresponds to an image generation model, and the multiple image generation models have the same model structure.

[0264] Illustratively, the image generation model includes a denoising network layer for performing denoising. Optionally, the denoising network layer includes at least one encoding network layer and at least one decoding network layer, with the encoding network layer and the decoding network layer corresponding one-to-one and configured to perform opposite feature size conversions. The encoding network layer is configured to gradually compress the feature size, while the decoding network layer is configured to gradually restore the feature size.

[0265] In some embodiments, a one-to-one correspondence between multiple denoising network layers corresponding to multiple image generation models is determined. In an exemplary embodiment, input feature representations of the multiple denoising network layers are determined based on the corresponding denoising network layers at different viewing angles.

[0266] Optionally, the noisy feature representation is processed by image generation models corresponding to multiple perspectives, and when the noisy feature representation is denoised by any denoising network layer in the image generation model, the input feature representation of the corresponding perspective is determined, thereby obtaining multiple input feature representations based on multiple perspectives.

[0267] In some embodiments, multiple input feature representations are input into an attention pooling layer between multiple denoising network layers having corresponding relationships.

[0268] Among them, the attention pooling layer is a network layer that pre-connects multiple denoising network layers with corresponding relationships.

[0269] In some embodiments, the three-dimensional shared information shared between the three-dimensional transformation matrices corresponding to the multiple input feature representations is extracted through an attention pooling layer.

[0270] Optionally, the attention pooling layer is used to perform back-projection processing on multiple input feature representations to obtain three-dimensional transformation matrices corresponding to the multiple input feature representations, and perform attention pooling on the multiple three-dimensional transformation matrices to obtain a volume feature representation. The volume feature representation is used to represent the three-dimensional shared information shared by the multiple three-dimensional transformation matrices.

[0271] Schematically, as shown in FIG6 , there is a correspondence between the decoding network layer 611 corresponding to view A and the decoding network layer 621 corresponding to view B, and there is a correspondence between the decoding network layer 612 corresponding to view A and the decoding network layer 622 corresponding to view B.

[0272] Determine the input feature representation a1 to be input to the decoding network layer 611, determine the input feature representation b1 to be input to the decoding network layer 621, input the input feature representation a1 and the input feature representation b1 to the attention pooling layer corresponding to the decoding network layer 611 and the decoding network layer 621, and obtain the three-dimensional shared information 1 corresponding to the decoding network layer 611 and the decoding network layer 621;

[0273] Similarly, the input feature representation a2 to be input to the decoding network layer 612 is determined, the input feature representation b2 to be input to the decoding network layer 622 is determined, the input feature representation a2 and the input feature representation b2 are input to the attention pooling layer corresponding to the decoding network layer 612 and the decoding network layer 622, and the three-dimensional shared information 2 corresponding to the decoding network layer 611 and the decoding network layer 622 is obtained.

[0274] In some embodiments, the attention pooling layer is further explained, as shown in FIG7 , which is a schematic diagram of the internal structure of the pooling layer of the attention mechanism.

[0275] Taking two perspectives (perspective A and perspective B) among multiple perspectives as an example, the current input feature representation of the coding network layer corresponding to perspective A is input feature representation 711; the current input feature representation of the coding network layer corresponding to perspective B is input feature representation 721, and there is a corresponding relationship between these two coding network layers.

[0276] Back-projection processing is performed on the input feature representation 711 of view A and the input feature representation 721 of view B, respectively, to obtain a three-dimensional transformation matrix 712 corresponding to the input feature representation 711 and a three-dimensional transformation matrix 722 corresponding to the input feature representation 721.

[0277] In addition, attention pooling is performed on the three-dimensional transformation matrix 712 and the three-dimensional transformation matrix 722 to obtain a volume feature representation 730, which is used to represent the three-dimensional shared information shared by multiple three-dimensional transformation matrices.

[0278] Step 550 : Adjust the multiple input feature representations using the three-dimensional shared information to obtain multiple adjusted feature representations.

[0279] There is a corresponding relationship between multiple input feature representations and multiple adjustment feature representations.

[0280] In some embodiments, as shown in Figure 7, after obtaining the volume feature representation 730 that represents the three-dimensional shared information between multiple three-dimensional transformation matrices, the volume feature representation 730 is coordinate mapped to map the volume feature representation 730 to the camera coordinate system of the corresponding perspective, and obtain the coordinate feature representation 741 corresponding to perspective A and the coordinate feature representation 742 corresponding to perspective B.

[0281] In addition, attention pooling operations in the projection dimension are performed on the coordinate feature representation 741 and the coordinate feature representation 742 respectively to obtain a three-dimensional feature representation 751 corresponding to perspective A and a three-dimensional feature representation 752 corresponding to perspective B; then, the residual feature representation corresponding to perspective A is output based on the three-dimensional feature representation 751, and the residual feature representation corresponding to perspective B is output based on the three-dimensional feature representation 752.

[0282] Step 560 : generating perspective images corresponding to the entity element at multiple perspectives based on the multiple adjusted feature representations.

[0283] In schematic form, after obtaining multiple adjusted feature representations corresponding to multiple input feature representations, the input feature representation is replaced by the adjusted feature representation and input into the denoising network layer corresponding to the input feature representation, thereby performing feature size conversion and / or denoising processing on the adjusted feature representation containing three-dimensional shared information through the denoising network layer, thereby avoiding the problem of lack of correlation when independent analysis is performed through the denoising network layers corresponding to different perspectives.

[0284] Among them, multiple perspective images are used to integrate and generate a three-dimensional model representing the entity elements.

[0285] Schematically, multiple perspective images are image contents that display entity elements from different perspectives. After obtaining the multiple perspective images, the multiple perspective images are processed with the help of a preset program, so that the multiple perspective images are integrated to generate a three-dimensional model representing the entity elements. The three-dimensional model can display the entity elements from multiple perspectives, and the shape, structure and other attribute information of the three-dimensional model conforms to the multiple perspective images.

[0286] In an optional embodiment, denoising is performed on the adjusted feature representations of the nth denoising network layer corresponding to multiple perspectives to obtain input feature representations of the n+1th denoising network layer corresponding to multiple perspectives.

[0287] Wherein, n is a positive integer not greater than m. Illustratively, based on the above process of obtaining the adjusted feature representation, after obtaining the adjusted feature representation obtained by adjusting the input feature representation of the nth denoising network layer, the adjusted feature representation is denoised by the nth denoising network layer, thereby obtaining the input feature representation to be input to the (n+1)th denoising network layer.

[0288] In some embodiments, in response to passing through the mth denoising network layer, denoising feature representations of the mth denoising network layer output corresponding to multiple perspectives are obtained.

[0289] Indicatively, the m-th denoising network layer is the last denoising network layer. After the adjusted feature representation after adjusting the input feature representation corresponding to the m-th denoising network layer passes through the m-th denoising network layer, the denoised feature representation is obtained; based on the participation of multiple perspectives in the above process, the denoised feature representations output by the m-th denoising network layer corresponding to the multiple perspectives are obtained.

[0290] Schematically, as shown in FIG6, after the mth denoising network layer, the denoising feature representation of the mth denoising network layer output corresponding to multiple perspectives is obtained, thereby converting z tDenoising is z t+1 , when no noise processing is done, z t is the noise feature representation; after the denoising process, z t is the noise feature representation of the middle layer, z t+1 is the denoising feature representation, and is also the denoising feature representation of the intermediate layer that will undergo the next iterative denoising process; z t+1 After being represented as a noise feature, z t+2 Based on z t+1 The obtained denoising feature representation, etc.

[0291] In some embodiments, iterative denoising processing is performed based on the denoising feature representation at multiple viewing angles until the number of iterations is reached, thereby obtaining decoding feature representations corresponding to the multiple viewing angles.

[0292] Indicatively, each view is corresponded to a denoising feature representation and then subjected to an iterative denoising process through the above denoising process, and a decoded feature representation is obtained after reaching this iteration.

[0293] The decoded feature representation is used to represent the feature representation obtained after denoising the noisy feature representation.

[0294] In some embodiments, the plurality of decoded feature representations are passed through decoders respectively to generate perspective images corresponding to the entity element at a plurality of perspectives.

[0295] Schematically, the decoder is a decoding layer configured in the image generation model corresponding to the viewpoint, which can decode the decoded feature representation to generate a viewpoint image corresponding to the viewpoint; or, the decoder is a decoding layer outside the image generation model, which can decode the decoded feature representation to generate a viewpoint image corresponding to the viewpoint, etc.

[0296] In summary, in the embodiments of the present application, a process is introduced in which the image generation data is used as a guiding condition for denoising, and the noisy feature representation is denoised at multiple perspectives. The image generation data is used as a denoising condition to guide the adjustment of the noisy feature representation during the denoising process, and the input feature representation is replaced by the adjusted feature representation, and denoising is performed through a denoising network layer. When there are multiple denoising network layers, the multiple denoising network layers can be used to perform denoising in sequence, thereby generating perspective images corresponding to the entity elements at multiple perspectives, thereby obtaining a three-dimensional model with stronger consistency and higher perspective correlation, thereby improving the authenticity of the generated three-dimensional model.

[0297] In an optional embodiment, the three-dimensional model generation method is called "using the attention mechanism of multiple visual dimensions and depth dimensions to construct a 3D consistent multi-view generation diffusion model", wherein the denoising process is performed by a stable diffusion model (Stable Diffusion).

[0298] Schematically, it is improved based on the diffusion generation model (diffusion model).

[0299] The traditional diffusion-based generative model takes a specified text as input and outputs an image that matches the text description. As shown in Figure 8, the core of the traditional diffusion model is a denoising autoencoder 810 that is called in a loop. The initial input is a Gaussian noise image. Each denoising step brings the image closer to the generated image until the noise is completely removed, generating an image that matches the user's input text description.

[0300] The embodiment of the present application is based on an improvement of the traditional diffusion generation model, so that multiple diffusion generation models can generate multiple perspective images of an object from different angles that meet the text description in parallel through text input, where each diffusion model generates an image from a viewpoint (perspective), and these perspective images have geometric consistency, as shown in Figure 6 above.

[0301] In the model structure shown in Figure 6, denoising autoencoders are called in parallel to generate images from multiple viewpoints. To ensure image consistency, an attention-based pooling module (e.g., AtnPool) is added. Its input is the feature image of the decoder portion of the denoising autoencoder from all viewpoints (the input feature representation described above), and its output is the residual feature representation of these viewpoint images. By adding the residual feature representation back to the input feature representation, the denoising autoencoder achieves three-dimensional consistency across all viewpoints.

[0302] Schematically, the core of the three-dimensional model generation method is to insert a module based on multi-viewpoint three-dimensional expression into multiple parallel traditional diffusion generation model pipelines. This module can be called the pooling module of the attention mechanism (that is, the pooling layer of the attention mechanism mentioned above), and the pooling module of the attention mechanism is used to ensure consistency between each other.

[0303] Among them, the input of the pooling module of the attention mechanism is the feature image of each diffusion model (the input feature representation mentioned above). The pooling module projects the feature images under different viewpoints into three-dimensional space and fuses them into a unified feature volume expression (volume feature representation) through the attention mechanism. The expression is then reprojected to each viewpoint through the attention mechanism, and the residual of the consistent feature image is output (residual feature representation).

[0304] As shown in Figure 7, this is a schematic diagram of the internal structure of the pooling layer of the attention mechanism.

[0305] Among them, the input of the pooling layer of the attention mechanism is the feature image of the denoising autoencoder for each viewpoint (i.e., the input feature representation mentioned above); first, the feature image is projected back into the 3D space through back projection, and the input feature representation of the two-dimensional scale is converted into a feature volume expression of the three-dimensional scale (i.e., the three-dimensional transformation matrix mentioned above). Then, in the multi-view dimension, the attention pooling mechanism is used to normalize the feature volumes under all viewpoints into a unified feature volume expression (i.e., the volume feature representation that characterizes the three-dimensional shared information is obtained); then, the unified volume feature representation is remapped back to the coordinate system of each viewpoint (i.e., the camera coordinate system corresponding to the viewing angle) through coordinate mapping; then, through the attention pooling mechanism on the depth dimension of the projection, the 3D feature volume (volume feature representation) is reprojected back to the 2D feature image (i.e., the coordinate feature representation mentioned above); the final output is a feature residual image (residual feature representation) of the same size as the input.

[0306] Optionally, the following content describes the “multiview attention pooling” process and the “ray attention pooling” process as shown in FIG7 .

[0307] (1) Attention Pooling in Multi-View Dimensions

[0308] Schematically, for a generation task with N viewpoints, the viewpoints are numbered i=1…N, and the input of the multi-view attention pooling module is N feature images (input feature representation), each of which is a feature image of the decoder part of the denoising autoencoder. The feature image of viewpoint i is denoted as m i , first back-project it into the three-dimensional transformation matrix v in three-dimensional space i , as shown in Formula 1 above.

[0309] Afterwards, the volume representation of N viewpoints is unified into one volume representation through the attention pooling mechanism - volume feature representation As shown in formula 2 above.

[0310] It is worth noting that in the pooling layer of the attention mechanism, during the training process, the parameters that can be optimized are the parameters of the attention layer.

[0311] (2) Attention Pooling in the Projection Dimension

[0312] Schematically, this is responsible for representing the unified volume feature Reproject the feature image back (such as the residual feature representation mentioned above) so that the feature image after reprojection records the feature information with three-dimensional geometric consistency, and add the feature image m of each viewpoint input back in the form of residuali (Input feature representation), and obtain the denoised feature representation as described above.

[0313] The first step is to express the volume feature Through coordinate mapping, the difference is expressed as the volume under the camera coordinates corresponding to each viewpoint The three-dimensional feature representation is shown in Formula 3 above.

[0314] Then each 3D feature representation is represented by attention pooling in the projection dimension Projected into 2D feature image As shown in formula 4 above.

[0315] Optionally, It will be transformed into an MLP and input feature representation m i The residual feature representation with the same feature size is added back to the corresponding input feature representation of the denoising autoencoder.

[0316] Among them, the initial values ​​of the parameters of the MLP multilayer perceptron are set to all zero.

[0317] It is worth noting that in the attention pooling process of the projection dimension, during the training process, the parameters that can be optimized are the parameters of the attention layer and the parameters of the MLP.

[0318] In some embodiments, the training process is briefly described. For example, a diffusion model corresponding to each viewpoint is used as an image generation model. Similar to a traditional diffusion model, when training the diffusion model corresponding to each viewpoint, relatively independent noise is added to the image at each viewpoint for training.

[0319] Schematically, the essence of the diffusion model training process is to add noise to the source image P, perform a denoising process with the help of the latent space in the diffusion model, obtain the target image P' after denoising, and compare the differences between the source image P and the target image P' to train the diffusion model.

[0320] As shown in Figure 8, in the diffusion model, by cyclically calling the denoising autoencoder 810, the traditional diffusion model performs a denoising process based on the input feature representation, and continuously removes the noise in the input feature representation until an image that meets the user input text description is generated; in the process provided by the embodiment of the present application as shown in Figure 6, although there is also a process of cyclically calling the denoising autoencoder, the denoising process is a denoising process based on the adjusted feature representation after adjusting the input feature representation, and the adjusted feature representation has good geometric consistency; by continuously removing the noise in the adjusted feature representation, an image that meets the user input text description is generated.

[0321] Alternatively, you can selectively fix the model parameters in the diffusion model and only train the aforementioned model parameters. For example, for the multi-viewpoint attention pooling process, only optimize the parameters of the attention layer; for the projection dimension attention pooling process, only optimize the parameters of the attention layer and / or the parameters of the MLP.

[0322] The optimization objective of the diffusion model remains to estimate the noise added at each step, just like the traditional diffusion model. Since the noise added to the source image at each viewpoint is independent, the attention pooling module can be trained separately for each viewpoint during training, saving video memory.

[0323] In the model training process, the noise added to the source image at each viewpoint can be the same or different, which is not limited here.

[0324] In some embodiments, through small data sets and simple training (such as hundreds to thousands of iterations of training), the generators under different viewpoints are made to meet geometric consistency.

[0325] Schematically, as shown in FIG9 , the result of generating a three-dimensional model using the three-dimensional model generation method proposed in this solution is compared with the result generated by Zero123 in the related art.

[0326] It shows examples under three viewing angles (viewpoints), namely viewpoint 1, viewpoint 2 and viewpoint 3.

[0327] In area 910 shown in Figure 9, the entity element is an identification symbol, which is a three-dimensional element. Compared with the true value, in the results generated by Zero123 in the related art, viewpoints 1 and 2 still have a certain degree of similarity, but viewpoint 3 is completely distorted. In contrast, using the three-dimensional model generation method proposed in this solution, whether it is viewpoint 1, viewpoint 2 or viewpoint 3, there is a high degree of similarity with the true value, thereby being able to restore a more accurate three-dimensional model.

[0328] Similarly, in area 920 shown in Figure 9, the entity element is a ship, which is a three-dimensional element. Compared with the true value, in the results generated by Zero123 in the related art, although viewpoints 1, 2 and 3 can roughly restore the shape of the ship, the difference with the true value is still large. In contrast, using the three-dimensional model generation method proposed in this solution, whether it is viewpoint 1, viewpoint 2 or viewpoint 3, there is a high degree of similarity with the true value, thereby being able to restore a more accurate three-dimensional model.

[0329] Similarly, in area 930 shown in Figure 9, the entity element is an oil barrel, which is a three-dimensional element. Compared with the true value, in the results generated by Zero123 in the related art, viewpoint 1 still has a certain similarity, but viewpoints 2 and 3 are still quite different from the true value. In contrast, using the three-dimensional model generation method proposed in this solution, whether it is viewpoint 1, viewpoint 2 or viewpoint 3, there is a high degree of similarity with the true value, thereby being able to restore a more accurate three-dimensional model.

[0330] That is, taking the true value as a reference, it can be noted that the results obtained by applying the 3D model generation method proposed in this scheme are compared with the results obtained by Zero123, and it is observed that the 3D consistency and material consistency of multiple views are significantly improved.

[0331] In some embodiments, as shown in FIG10 , the entity element is a virtual warrior, and the input image 1010 is used as the image generation data, that is, the input image 1010 is used as the denoising condition. After applying the above-mentioned image generation method for processing, a perspective image 1020 of another perspective is obtained; after processing by the Zero123 method, a perspective image 1030 of the same perspective as the perspective image 1020 is obtained. It can be clearly seen that the perspective image predicted by the Zero123 method is quite different from the input image 1010.

[0332] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0333] In summary, by comprehensively analyzing multiple input feature representations, the same information of multiple input feature representations on a three-dimensional scale can be obtained, thereby extracting three-dimensional shared information. The input feature representation is constrained with the help of three-dimensional shared information, thereby avoiding the problem of fragmentation when different perspectives are independently denoised through the corresponding denoising network layer. Adjusting the input feature representation through three-dimensional shared information is conducive to strengthening the correlation during the denoising process of different perspectives, thereby making the perspective images have a stronger correlation, which is conducive to generating a three-dimensional model with stronger geometric consistency based on the integration of multiple perspective images, and improving the authenticity and meticulousness of the entity elements represented by the three-dimensional model.

[0334] FIG11 is a block diagram of a three-dimensional model generation device provided by an exemplary embodiment of the present application. As shown in FIG11 , the device includes the following parts:

[0335] An acquisition module 1110 is configured to acquire a noise feature representation corresponding to the noise data, wherein the noise feature representation is used for denoising at multiple viewing angles to obtain viewing angle images corresponding to the entity elements at the multiple viewing angles.

[0336] a determination module 1120 for determining, when denoising the noisy feature representation using the denoising network layers corresponding to the multiple perspectives, input feature representations of the denoising network layers corresponding to the multiple perspectives, the input feature representations being feature representations input to the denoising network layers;

[0337] An extraction module 1130 is configured to obtain three-dimensional shared information shared by three-dimensional transformation matrices corresponding to a plurality of input feature representations, wherein the three-dimensional transformation matrix is ​​a matrix obtained by transforming the dimensions of the input feature representations;

[0338] An adjustment module 1140 is configured to adjust the plurality of input feature representations based on the three-dimensional shared information to obtain a plurality of adjusted feature representations, wherein a corresponding relationship exists between the plurality of input feature representations and the plurality of adjusted feature representations;

[0339] The generating module 1150 is configured to generate perspective images corresponding to the entity element at the multiple perspectives based on the multiple adjusted feature representations, and the multiple perspective images are integrated to generate a three-dimensional model representing the entity element.

[0340] In an optional embodiment, the determination module 1120 is also used to obtain image generation data, which is data collected for the physical element, and the image generation data is used to depict the physical element; using the image generation data as a denoising condition, the input feature representation of the denoising network layer corresponding to the multiple perspectives is determined, and the denoising condition is used to determine the noise prediction situation when denoising the noisy feature representation.

[0341] In an optional embodiment, the determination module 1120 is further used to obtain at least one image data collected for the entity element as the image generation data, where the image data is an image collected for the entity element at a preset perspective; or, to obtain text data used to describe the entity element as the image generation data.

[0342] In an optional embodiment, the extraction module 1130 is further used to respectively back-project the multiple input feature representations to obtain the three-dimensional transformation matrices corresponding to the multiple input feature representations; perform attention pooling processing on the multiple three-dimensional transformation matrices to obtain volume feature representations, and the volume feature representation is used to characterize the three-dimensional shared information shared between multiple three-dimensional transformation matrices.

[0343] In an optional embodiment, the extraction module 1130 is further used to respectively back-project the multiple input feature representations to obtain projection feature representations corresponding to the multiple perspectives; obtain parameter feature representations corresponding to the multiple perspectives, the parameter feature representations are feature representations obtained based on camera parameters corresponding to the perspectives, and the parameter feature representations are used to characterize spatial information under the perspective, wherein a correspondence exists between multiple parameter feature representations and multiple projection feature representations; based on the correspondence, the projection feature representation and the parameter feature representation under the same perspective are spliced ​​to obtain the three-dimensional transformation matrix corresponding to the multiple input feature representations.

[0344] In an optional embodiment, the extraction module 1130 is also used to obtain camera parameters corresponding to the multiple perspectives, the camera parameters including camera position and camera direction, the camera position being used to characterize the position of the camera relative to the physical element in the world coordinate system, and the camera direction being used to characterize the shooting direction of the camera relative to the physical element in the world coordinate system; obtaining parameter volume expressions corresponding to the multiple camera parameters, the parameter volume expression being a feature representation obtained by expressing the camera parameters in three-dimensional space, the parameter volume expression including perspective direction and perspective depth, the perspective direction being determined based on the direction of the voxel in the three-dimensional space relative to the camera center, and the perspective depth being determined based on the distance of the voxel relative to the camera center; encoding the parameter volume expression through a preset feature encoding function to obtain the parameter feature representation corresponding to the multiple perspectives.

[0345] In an optional embodiment, the extraction module 1130 is also used to determine the voxel sets represented by the multiple three-dimensional transformation matrices respectively, and the voxel sets are used to represent the sets of multiple voxels in the three-dimensional space when the three-dimensional transformation matrix is ​​obtained; determine multiple attention values ​​based on multiple voxels in the same voxel position in the multiple voxel sets; and pool the multiple attention values ​​to obtain the volume feature representation.

[0346] In an optional embodiment, the adjustment module 1140 is also used to obtain a volume feature representation that characterizes the three-dimensional shared information; based on the perspectives corresponding to the multiple input feature representations and the volume feature representation, three-dimensional feature representations corresponding to the multiple perspectives are obtained, and the three-dimensional feature representation is used to characterize the influence of the volume feature representation on the input feature representation in the spatial dimension, wherein there is a corresponding relationship between the multiple three-dimensional feature representations and the multiple input feature representations; based on the corresponding relationship, the multiple adjustment feature representations are obtained through the three-dimensional feature representation and the input feature representation under the same perspective.

[0347] In an optional embodiment, the adjustment module 1140 is also used to determine the camera coordinate systems corresponding to the multiple perspectives, where the camera coordinate system is a coordinate system established with the camera used when determining the perspective as a reference point; with the camera coordinate system as a reference, the volume feature representation is mapped to the three-dimensional space to obtain the coordinate feature representations corresponding to the multiple perspectives; based on the coordinate feature representations corresponding to the multiple perspectives, the three-dimensional feature representations corresponding to the multiple perspectives in the three-dimensional space are obtained.

[0348] In an optional embodiment, the adjustment module 1140 is also used to obtain the perspective depths respectively represented by the multiple perspectives; under the same perspective, based on the perspective depths and the coordinate feature representation, the three-dimensional feature representations corresponding to the multiple perspectives in the three-dimensional space are obtained.

[0349] In an optional embodiment, the adjustment module 1140 is also used to determine a voxel set represented by a three-dimensional transformation matrix corresponding to multiple perspectives, wherein the voxel set includes multiple voxels, and each voxel corresponds to a perspective depth; multiple voxels in the voxel set are filled with the perspective depth corresponding to the voxel as the voxel value to obtain a voxel block set with the same voxel value; based on the voxel block set and the coordinate feature representation, the three-dimensional feature representation corresponding to the multiple perspectives in the three-dimensional space is obtained.

[0350] In an optional embodiment, the adjustment module 1140 is also used to encode the voxel block set using a preset encoding function to obtain a voxel feature representation; at the same perspective, the voxel feature representation and the coordinate feature representation are spliced ​​to obtain the three-dimensional feature representation corresponding to the multiple perspectives in the three-dimensional space.

[0351] In an optional embodiment, the adjustment module 1140 is also used to project the three-dimensional feature representations corresponding to multiple perspectives into a two-dimensional space to obtain residual feature representations corresponding to multiple perspectives; at the same perspective, the residual feature representation and the input feature representation are spliced ​​to obtain the adjusted feature representations corresponding to multiple perspectives.

[0352] In an optional embodiment, each perspective corresponds to m denoising network layers, where m is a positive integer;

[0353] The adjustment module 1140 is also used to denoise the adjusted feature representations of the nth denoising network layer corresponding to the multiple perspectives, and obtain the input feature representations of the n+1th denoising network layer corresponding to the multiple perspectives, where n is a positive integer not greater than m; after the mth denoising network layer, obtain the denoised feature representations output by the mth denoising network layer corresponding to the multiple perspectives; and generate the perspective images corresponding to the entity elements at the multiple perspectives based on multiple denoising feature representations.

[0354] In an optional embodiment, the adjustment module 1140 is also used to iteratively denoise the denoised feature representation under the multiple perspectives until the number of iterations is reached, and obtain decoded feature representations corresponding to the multiple perspectives respectively, and the decoded feature representation is used to characterize the feature representation obtained after denoising the noisy feature representation; the multiple decoded feature representations are passed through the decoder respectively to generate the perspective images corresponding to the entity elements under the multiple perspectives respectively.

[0355] Figure 12 shows a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. The server 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. The server 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0356] The mass storage device 1206 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1206 and its associated computer-readable media provide non-volatile storage for the server 1200. In other words, the mass storage device 1206 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0357] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. The system memory 1204 and mass storage device 1206 described above may be collectively referred to as memory.

[0358] According to various embodiments of the present application, the server 1200 can also be connected to a remote computer on a network via a network such as the Internet. That is, the server 1200 can be connected to the network 1212 via a network interface unit 1211 connected to the system bus 1205. Alternatively, the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown). The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0359] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the three-dimensional model generation method provided by the above-mentioned method embodiments.

[0360] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the three-dimensional model generation method provided by the above-mentioned method embodiments.

[0361] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the three-dimensional model generation method described in any of the above embodiments.

[0362] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A three-dimensional model generation method, executed by a computer device, the method comprising: Acquire a noise feature representation corresponding to the noise data, wherein the noise feature representation is used to remove noise at multiple viewing angles to obtain viewing angle images corresponding to the entity elements at the multiple viewing angles respectively; When denoising the noisy feature representation through the denoising network layers corresponding to the multiple perspectives, respectively, determining the input feature representation of the denoising network layers corresponding to the multiple perspectives, respectively, the input feature representation being the feature representation input to the denoising network layer; Acquire three-dimensional shared information shared by three-dimensional transformation matrices corresponding to a plurality of input feature representations, wherein the three-dimensional transformation matrix is ​​a matrix obtained by transforming the dimensions of the input feature representations; Adjusting the plurality of input feature representations based on the three-dimensional shared information to obtain a plurality of adjusted feature representations, wherein there is a corresponding relationship between the plurality of input feature representations and the plurality of adjusted feature representations; Based on the multiple adjustment feature representations, perspective images corresponding to the entity element at the multiple perspectives are generated, and the multiple perspective images are used to integrate and generate a three-dimensional model representing the entity element.

2. The method according to claim 1, wherein: The step of determining the input feature representations of the denoising network layers corresponding to the multiple perspectives respectively includes: Acquire image generation data, where the image generation data is data collected for the entity element, and the image generation data is used to depict the entity element; The image generation data is used as a denoising condition to determine the input feature representations of the denoising network layers corresponding to the multiple perspectives respectively, and the denoising condition is used to determine the noise prediction when denoising the noisy feature representation.

3. The method according to claim 1 or 2, wherein: The step of obtaining image generation data comprises: Acquire at least one image data collected for the physical element as the image generation data, wherein the image data is an image collected for the physical element at a preset viewing angle; or, Text data for describing the entity element is acquired as the image generation data.

4. The method according to any one of claims 1 to 3, wherein: The obtaining of three-dimensional shared information shared by three-dimensional transformation matrices corresponding to the plurality of input feature representations includes: Back-projecting the plurality of input feature representations respectively to obtain the three-dimensional transformation matrices respectively corresponding to the plurality of input feature representations; Attention pooling is performed on multiple three-dimensional transformation matrices to obtain a volume feature representation, where the volume feature representation is used to characterize the three-dimensional shared information shared among the multiple three-dimensional transformation matrices.

5. The method according to any one of claims 1 to 4, wherein: The back-projecting the plurality of input feature representations respectively to obtain the three-dimensional transformation matrices respectively corresponding to the plurality of input feature representations includes: Back-projecting the plurality of input feature representations respectively to obtain projection feature representations corresponding to the plurality of viewing angles respectively; Acquire parameter feature representations corresponding to the multiple perspectives respectively, the parameter feature representations being feature representations obtained based on camera parameters corresponding to the perspectives, the parameter feature representations being used to characterize spatial information under the perspectives, wherein there is a corresponding relationship between the multiple parameter feature representations and the multiple projection feature representations; Based on the corresponding relationship, the projection feature representation and the parameter feature representation under the same viewing angle are spliced ​​to obtain the three-dimensional transformation matrix corresponding to the multiple input feature representations respectively.

6. The method according to any one of claims 1 to 5, wherein: The obtaining of parameter feature representations corresponding to the multiple perspectives respectively includes: Acquire camera parameters corresponding to the multiple perspectives respectively, the camera parameters including camera position and camera direction, the camera position is used to represent the position of the camera relative to the physical element in the world coordinate system, and the camera direction represents the shooting direction of the camera relative to the physical element in the world coordinate system; Obtain parameter volume expressions corresponding to multiple camera parameters respectively, wherein the parameter volume expression is a feature representation obtained by expressing the camera parameters in a three-dimensional space, and the parameter volume expression includes a viewing direction and a viewing depth. The direction of the voxel in the three-dimensional space relative to the center of the camera is determined, and the viewing depth is determined based on the distance of the voxel relative to the center of the camera; The parameter volume expression is encoded by a preset feature encoding function to obtain the parameter feature representations corresponding to the multiple viewing angles respectively.

7. The method according to any one of claims 1 to 6, wherein: The performing attention pooling processing on the multiple three-dimensional transformation matrices to obtain volume feature representation includes: Determine voxel sets respectively represented by the multiple three-dimensional transformation matrices, where the voxel sets are used to represent a set of multiple voxels in a three-dimensional space when the three-dimensional transformation matrices are obtained; determining a plurality of attention values ​​based on a plurality of voxels at a same voxel position in a plurality of voxel sets; The multiple attention values ​​are pooled to obtain the volume feature representation.

8. The method according to any one of claims 1 to 7, wherein: The adjusting the plurality of input feature representations based on the three-dimensional shared information comprises: Acquire a volume feature representation representing the three-dimensional shared information; Based on the viewing angles respectively corresponding to the multiple input feature representations and the volume feature representation, obtaining three-dimensional feature representations respectively corresponding to the multiple viewing angles, the three-dimensional feature representations being used to characterize the influence of the volume feature representation on the input feature representation in the spatial dimension, wherein there is a corresponding relationship between the multiple three-dimensional feature representations and the multiple input feature representations; Based on the corresponding relationship, the multiple adjusted feature representations are obtained through the three-dimensional feature representation and the input feature representation under the same viewing angle.

9. The method according to any one of claims 1 to 8, wherein: The obtaining of three-dimensional feature representations corresponding to the multiple perspectives respectively based on the perspectives respectively corresponding to the multiple input feature representations and the volume feature representations includes: Determine the camera coordinate systems corresponding to the multiple perspectives respectively, wherein the camera coordinate system is a coordinate system established by taking the camera used when determining the perspective as a reference point; Mapping the volume feature representation to a three-dimensional space with reference to the camera coordinate system to obtain coordinate feature representations corresponding to the multiple perspectives respectively; Based on the coordinate feature representations respectively corresponding to the multiple perspectives, the three-dimensional feature representations respectively corresponding to the multiple perspectives in the three-dimensional space are obtained.

10. The method according to any one of claims 1 to 9, wherein: The obtaining, based on the coordinate feature representations respectively corresponding to the multiple perspectives, the three-dimensional feature representations respectively corresponding to the multiple perspectives in the three-dimensional space comprises: Acquire the perspective depths respectively represented by the multiple perspectives; At the same viewing angle, based on the viewing angle depth and the coordinate feature representation, the three-dimensional feature representations corresponding to the multiple viewing angles in the three-dimensional space are obtained.

11. The method according to any one of claims 1 to 10, wherein: The obtaining, under the same viewing angle, based on the viewing angle depth and the coordinate feature representation, the three-dimensional feature representations corresponding to the multiple viewing angles in the three-dimensional space respectively includes: Determine a voxel set represented by a three-dimensional transformation matrix corresponding to a plurality of viewing angles, wherein the voxel set includes a plurality of voxels, and each voxel corresponds to a viewing angle depth; Filling a plurality of voxels in the voxel set with a viewing angle depth corresponding to the voxel as a voxel value to obtain a voxel block set having the same voxel value; Based on the voxel block set and the coordinate feature representation, the three-dimensional feature representations corresponding to the multiple perspectives in the three-dimensional space are obtained.

12. The method according to any one of claims 1 to 11, wherein: The obtaining, based on the voxel block set and the coordinate feature representation, the three-dimensional feature representations corresponding to the multiple perspectives in the three-dimensional space respectively includes: Encoding the voxel block set using a preset encoding function to obtain a voxel feature representation; At the same viewing angle, the voxel feature representation and the coordinate feature representation are spliced ​​to obtain the three-dimensional feature representations corresponding to the multiple viewing angles in the three-dimensional space.

13. The method according to any one of claims 1 to 12, wherein: Based on the corresponding relationship, the three-dimensional feature representation and the input feature representation under the same viewing angle are spliced ​​to obtain the multiple adjusted feature representations: Projecting the three-dimensional feature representations corresponding to the multiple perspectives into a two-dimensional space to obtain residual feature representations corresponding to the multiple perspectives; At the same viewing angle, the residual feature representation and the input feature representation are concatenated to obtain the adjusted feature representations corresponding to multiple viewing angles respectively.

14. The method according to any one of claims 1 to 13, wherein: Each view has m denoising network layers corresponding to it, where m is a positive integer; The generating the perspective images corresponding to the entity elements under the multiple perspectives respectively based on the multiple adjustment feature representations includes: Denoising the adjusted feature representations of the nth denoising network layer corresponding to the multiple perspectives, to obtain input feature representations of the n+1th denoising network layer corresponding to the multiple perspectives, where n is a positive integer not greater than m; After the mth denoising network layer, obtaining denoising feature representations output by the mth denoising network layer corresponding to the multiple perspectives respectively; The perspective images corresponding to the entity elements at the multiple perspectives are generated based on multiple denoising feature representations.

15. The method according to any one of claims 1 to 14, wherein: The generating the perspective images corresponding to the entity elements at the multiple perspectives respectively based on the multiple denoising feature representations includes: Iteratively denoising the denoised feature representation under the multiple viewing angles until the number of iterations is reached, thereby obtaining decoded feature representations corresponding to the multiple viewing angles, respectively, the decoded feature representations being used to represent feature representations obtained after denoising the noisy feature representation; The multiple decoded feature representations are passed through decoders respectively to generate the viewing angle images corresponding to the entity elements under the multiple viewing angles respectively.

16. A three-dimensional model generating device, the device comprising: An acquisition module, used to acquire a noise feature representation corresponding to the noise data, wherein the noise feature representation is used to remove noise at multiple viewing angles to obtain viewing angle images corresponding to the entity elements at the multiple viewing angles; A determination module, configured to determine, when denoising the noisy feature representation through the denoising network layers corresponding to the multiple perspectives, input feature representations of the denoising network layers corresponding to the multiple perspectives, respectively, wherein the input feature representations are feature representations input to the denoising network layers; An extraction module, used to obtain three-dimensional shared information shared by three-dimensional transformation matrices corresponding to a plurality of input feature representations, wherein the three-dimensional transformation matrix is ​​a matrix obtained by transforming the dimension of the input feature representation; an adjustment module, configured to adjust the plurality of input feature representations based on the three-dimensional shared information to obtain a plurality of adjusted feature representations, wherein a corresponding relationship exists between the plurality of input feature representations and the plurality of adjusted feature representations; A generation module is used to generate perspective images corresponding to the entity element under the multiple perspectives based on the multiple adjustment feature representations, and the multiple perspective images are used to integrate and generate a three-dimensional model representing the entity element.

17. A computer device, comprising a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the three-dimensional model generation method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, wherein at least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to implement the three-dimensional model generation method according to any one of claims 1 to 15.

19. A computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the three-dimensional model generation method according to any one of claims 1 to 15 is implemented.

Images