Three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The present disclosure relates to a 3D reconstruction method, namely, an apparatus, a device, namely, a storage medium, and the method comprises: inputting a source image into a known multi-view 3D reconstruction network, and outputting a video and camera pose information corresponding to the source image; extracting each image frame of the video, and matching each extracted image frame with the camera pose information; identifying the source image to obtain a text description related to the source image; using the text description related to the source image and each image frame and camera pose information that match each other as inputs of a known image generation model, and generating images of various angles corresponding to the source image to form a 3D model corresponding to the source image, so as to accurately reconstruct objects in the image in 3D.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and apparatus, device and storage medium. Background Art

[0002] Computer vision is an important research topic in the field of artificial intelligence. Traditional digital image processing algorithms such as edge detection, Hough transform, filtering, etc. remain at the level of human brain’s understanding of images, and digital image processing methods are all in line with human cognition. With the advent of deep learning, people have gradually realized the role of deep neural networks in image processing, especially the emergence of AlexNet, which has pointed out the development direction for future computer vision and even the entire field of artificial intelligence. Nowadays, large models have been released to the public one after another. Their effects and influence are obvious to all. They have greatly improved the work efficiency of some people and brought profound social influence. For example, the emergence of Segment Anything Model has basically drawn a semicolon for the field of semantic segmentation in computer vision. Artificial intelligence has developed rapidly, and gradually, the field of computer vision has also made some achievements in three-dimensional vision. Three-dimensional reconstruction or inverse rendering is one of the important topics.

[0003] The emergence of deep neural networks not only affects tasks such as segmentation and classification, but also derives a new way to describe 3D models, NeRF (Neural Radiance Fields). NeRF itself is a deep neural network that fits a mapping from camera coordinate pose to RGB information. When a NeRF is trained, all 3D model information is stored in NeRF as parameters of the deep neural network. It can also be said that NeRF itself is a 3D model, but its carrier form is a deep neural network. The biggest advantage of using NeRF to describe a 3D model is that it is compatible with other deep learning algorithms. Compared with traditional 3D model representation methods, NeRF can directly accept feature information extracted by deep neural networks and use it to modify the 3D model stored inside it, which gives NeRF great generalization performance in the algorithm.

[0004] The diffusion model is a generative model that has been rigorously mathematically proven. It is widely used in current image generation networks and is the underlying algorithm and basic tool for many large models. It can generate a picture that matches the text description based on text prompt information. It is widely used in current image generation networks and is the underlying algorithm and basic tool for many large models. It can generate a picture that matches the text description based on text prompt information. This has also spawned a very direct method of generating three-dimensional models, which is to construct a three-dimensional model based on the text descriptions from various angles and the pictures they generate.

[0005] However, the above two 3D reconstruction methods usually generate 3D models based on text and are not suitable for robots that need to reconstruct 3D models based on pictures. Summary of the invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a three-dimensional reconstruction method and apparatus, a device and a storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction method, comprising:

[0008] Input the source image into the known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image;

[0009] Extracting each image frame of the video, and matching each extracted image frame with camera pose information;

[0010] Identify the source image and obtain text description related to the source image;

[0011] The text description related to the source image and each matching image frame and camera pose information are used as input to a known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image.

[0012] In a possible implementation, the source image is input into a known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image, including:

[0013] The source image is input into the known multi-view Figure 3 3D reconstruction network to generate a target 3D model corresponding to the source image;

[0014] Collect images at different angles around the target 3D model and calculate the camera pose at each acquisition angle;

[0015] A video corresponding to the source image is generated according to an image sequence formed by images at different angles.

[0016] In a possible implementation manner, identifying the source image to obtain a text description related to the source image includes:

[0017] The pre-trained image recognition model is used to identify the source image and obtain text descriptions related to the source image.

[0018] In a possible implementation, the image generation model includes a diffusion model and a three-dimensional reconstruction unit, and the text description related to the source image and each image frame and camera pose information that match each other are used as inputs of the known image generation model to generate images at various angles corresponding to the source image to form a three-dimensional model corresponding to the source image, including:

[0019] Input each image frame and camera pose information that matches each other into a 3D reconstruction unit to obtain an initial 3D reconstruction unit;

[0020] Inputting a text description related to a source image into a known diffusion model to generate a first image corresponding to the source image;

[0021] Generate a rendering using an initial 3D reconstruction unit;

[0022] Training an initial 3D reconstruction unit according to the first image and the rendering to obtain a known 3D reconstruction unit;

[0023] Based on the known three-dimensional reconstruction unit, a three-dimensional model corresponding to the source image is obtained.

[0024] In a possible implementation manner, inputting the text description related to the source image into a known diffusion model to generate a first image corresponding to the source image includes:

[0025] Mapping text descriptions associated with source images into embedding vectors for generated images;

[0026] A first image corresponding to the source image is generated according to the embedding vector.

[0027] In a possible implementation, the training of an initial 3D reconstruction unit according to the first image and the rendering image to obtain a known 3D reconstruction unit includes:

[0028] Using a known diffusion model, gradually adding noise to the rendering image to obtain a first Gaussian noise corresponding to the rendering image;

[0029] Denoising the first Gaussian noise using the diffusion model used to generate the first image to obtain a second image;

[0030] The noise between the second image and the first image is used as the second Gaussian noise;

[0031] Calculate the noise loss according to the rendering image, the first Gaussian noise and the second image;

[0032] When the noise loss is less than a preset threshold, the initial 3D reconstruction unit is used as a known 3D reconstruction unit;

[0033] When the noise loss is greater than or equal to the preset threshold, the parameters of the initial 3D reconstruction unit are adjusted, and the noise loss is recalculated for the 3D reconstruction unit after the parameter adjustment until the noise loss is less than the preset threshold.

[0034] In a possible implementation manner, the recalculating the noise loss for the parameter-adjusted 3D reconstruction unit includes:

[0035] Returning to the step of generating a first image corresponding to the source image;

[0036] In the case that the input of the diffusion model is a text description when the first image was generated last time, the first image is generated by using the image as the input of the known diffusion model, wherein the image is selected from a set including a source image and an input 3D reconstruction unit image frame;

[0037] When the input of the diffusion model is an image when the first image is generated last time, the first image is generated using the text description as the input of the known diffusion model.

[0038] In a second aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction device, including:

[0039] Input module, used to input source images into known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image;

[0040] A matching module, used for extracting each image frame of the video and matching each extracted image frame with camera pose information;

[0041] A recognition module, used to recognize the source image and obtain a text description related to the source image;

[0042] The generation module is used to use the text description related to the source image and each matching image frame and camera pose information as input to the known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image.

[0043] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0044] Memory, used to store computer programs;

[0045] The processor is used to implement the above-mentioned three-dimensional reconstruction method when executing the program stored in the memory.

[0046] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned three-dimensional reconstruction method when executed by a processor.

[0047] The above technical solution provided by the embodiment of the present disclosure has at least some or all of the following advantages compared with the prior art:

[0048] The three-dimensional reconstruction method described in the embodiment of the present disclosure is to input the source image into a known multi-view Figure 3 A three-dimensional reconstruction network is provided to output a video and camera pose information corresponding to a source image; each image frame of the video is extracted, and each extracted image frame is matched with the camera pose information; the source image is recognized to obtain a text description related to the source image; the text description related to the source image and each image frame and camera pose information that match each other are used as inputs of a known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image, so as to accurately reconstruct objects in the image in three dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0051] Figure 1 A schematic diagram of a three-dimensional reconstruction method according to an embodiment of the present disclosure is shown;

[0052] Figure 2 A schematic diagram of a three-dimensional reconstruction method according to another embodiment of the present disclosure is shown;

[0053] Figure 3 A schematic diagram of a detailed process of generating a final three-dimensional model from one image according to an embodiment of the present disclosure is shown;

[0054] Figure 4 A schematic diagram schematically shows a comparison of the three-dimensional reconstruction effects between the three-dimensional reconstruction method according to an embodiment of the present disclosure and the full-angle unsupervised learning framework LEGO model;

[0055] Figure 5 A schematic diagram of a structure of a three-dimensional reconstruction device according to an embodiment of the present disclosure is shown; and

[0056] Figure 6 The structural block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0058] See also Figure 1 , an embodiment of the present disclosure provides a three-dimensional reconstruction method, comprising the following steps:

[0059] S1, the source image is input into the known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image.

[0060] S2, extracting each image frame of the video, and matching each extracted image frame with camera pose information.

[0061] S3, identifying the source image and obtaining a text description related to the source image.

[0062] S4, taking the text description related to the source image and each image frame and camera pose information that matches each other as inputs to a known image generation model, generates images at various angles corresponding to the source image to form a three-dimensional model corresponding to the source image.

[0063] In this embodiment, in step S1, the source image is input into a known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image, including:

[0064] The source image is input into the known multi-view Figure 3 3D reconstruction network to generate a target 3D model corresponding to the source image;

[0065] Collect images at different angles around the target 3D model and calculate the camera pose at each acquisition angle;

[0066] A video corresponding to the source image is generated according to an image sequence formed by images at different angles.

[0067] In this embodiment, the known multi-view Figure 3The dimensional reconstruction network can be PixelNeRF, a multi-view NeRF (Neural Radiance Fields) based Figure 3 Reconstruct the network.

[0068] In this embodiment, the source image can be converted to an image size that is suitable for PixelNeRF, and a video about the image can be output through PixelNeRF, which contains PixelNeRF's speculation on the 3D model that the image may belong to. Unlike NeRF itself, which stores 3D models as a container, PixelNeRF directly constructs a mapping, the output of which is the camera coordinates and pose, and the output is the corresponding rendering under the coordinates and pose, so the method of generating videos is convenient for switching between these two 3D model expressions.

[0069] In this embodiment, the video generated by PixelNeRF includes the observation angle of the corresponding video frame, that is, the corresponding camera pose information. The video is extracted and each frame is matched with the relevant camera pose as a data set for training NeRF. The data set generated by PixelNeRF is used for initializing NeRF.

[0070] In this embodiment, in step S3, the source image is identified to obtain a text description related to the source image, including:

[0071] The source image is recognized using a pre-trained image recognition model to obtain a text description related to the source image, wherein the image recognition model is used to extract semantic information of the object shown in the source image from the source image.

[0072] In this embodiment, in step S4, the image generation model includes a diffusion model and a 3D reconstruction unit, and the text description related to the source image and each image frame and camera pose information that match each other are used as inputs of the known image generation model to generate images of various angles corresponding to the source image to form a 3D model corresponding to the source image, including:

[0073] Input each image frame and camera pose information that matches each other into a 3D reconstruction unit to obtain an initial 3D reconstruction unit, wherein the initial 3D reconstruction unit may be NeRF. This step may align the expression of the 3D object by PixelNeRF and NeRF;

[0074] Inputting a text description related to a source image into a known diffusion model to generate a first image corresponding to the source image;

[0075] Generate a rendering using an initial 3D reconstruction unit;

[0076] Training an initial 3D reconstruction unit according to the first image and the rendering to obtain a known 3D reconstruction unit;

[0077] Based on the known three-dimensional reconstruction unit, a three-dimensional model corresponding to the source image is obtained.

[0078] In this embodiment, inputting the text description related to the source image into a known diffusion model to generate a first image corresponding to the source image includes:

[0079] Mapping text descriptions associated with the source image into embedding vectors for generating the image using the trained CLIP model, wherein the embedding vectors are distributed in a latent space for generating the image, which is used to guide the latent space in which the diffusion model works;

[0080] Based on a known diffusion model, a first image corresponding to the source image is generated according to the embedding vector.

[0081] In this embodiment, the training of the initial 3D reconstruction unit according to the first image and the rendering image to obtain the known 3D reconstruction unit includes:

[0082] Using a known diffusion model, gradually adding noise to the rendering image to obtain a first Gaussian noise corresponding to the rendering image;

[0083] Denoising the first Gaussian noise using the diffusion model used to generate the first image to obtain a second image;

[0084] The noise between the second image and the first image is used as the second Gaussian noise;

[0085] Calculating a noise loss according to the rendering, the first Gaussian noise, and the second image, wherein the noise loss is obtained according to the noise between the first Gaussian noise and the rendering and the noise between the first Gaussian noise and the second image;

[0086] When the noise loss is less than a preset threshold, the initial 3D reconstruction unit is used as a known 3D reconstruction unit;

[0087] When the noise loss is greater than or equal to the preset threshold, the parameters of the initial 3D reconstruction unit are adjusted, and the noise loss is recalculated for the 3D reconstruction unit after the parameter adjustment until the noise loss is less than the preset threshold.

[0088] In this embodiment, the recalculating the noise loss for the parameter-adjusted 3D reconstruction unit includes:

[0089] Returning to the step of generating a first image corresponding to the source image;

[0090] In the case that the input of the diffusion model is a text description when the first image was generated last time, the first image is generated by using the image as the input of the known diffusion model, wherein the image is selected from a set including a source image and an input 3D reconstruction unit image frame;

[0091] When the input of the diffusion model is an image when the first image is generated last time, the first image is generated using the text description as the input of the known diffusion model.

[0092] In this embodiment, using an image as an input of a known diffusion model to generate a first image includes:

[0093] The image is distributed in a latent space for generating a target image, and based on a known diffusion model, a first image corresponding to the source image is generated according to the latent space where the image information is distributed.

[0094] like Figure 2 and Figure 3 As shown, with the known multi-view Figure 3 Taking PixelNeRF as an example, the 3D reconstruction network is used to describe the 3D reconstruction method of the present invention: an image with an apple drawn on it is taken as a source image and input into PixelNeRF to generate a video related to the apple, the video is framed to obtain multiple apple images and the posture corresponding to each apple image, the apple image and its corresponding posture are input into NeRF, NeRF is trained to obtain an initial 3D reconstruction unit, the source image is recognized to obtain a description "apple" related to the apple, "apple" is input into a diffusion model to generate a first image corresponding to the apple source image; an apple rendering is generated using the initial 3D reconstruction unit; text and images are used as alternating inputs of the diffusion model, and the initial 3D reconstruction unit is trained according to the first image and the apple rendering to obtain a known 3D reconstruction unit; based on the known 3D reconstruction unit, a 3D model corresponding to the source image is obtained.

[0095] The 3D reconstruction method disclosed in the present invention is compared with the LEGO model of the full-angle unsupervised learning framework for 3D reconstruction. Figure 4 As shown, the data of a certain angle range was cleaned before comparison, showing the optimization effect of the three-dimensional reconstruction method disclosed in the present invention on the LEGO neural network model of the cleaned angles.

[0096] The three-dimensional reconstruction method disclosed in the present invention utilizes multi-view Figure 3The 3D reconstruction network generates a video corresponding to the source image as the initialization parameter of the 3D reconstruction unit. Combined with the text description of the source image and the initialized 3D reconstruction unit, it can accurately generate the relevant 3D model based on a single or a small number of pictures, avoiding the problem of pre-labeling in traditional reconstruction methods. During the reconstruction, there is no need to input a large amount of calibrated image data, which greatly saves the preprocessing and labeling costs; it uses pre-trained diffusion models, and can utilize these models based on semantic information trained based on a large amount of data; it can generate an approximate 3D model of the object shown in the picture with only one picture.

[0097] See also Figure 5 , an embodiment of the present disclosure provides a three-dimensional reconstruction device, comprising:

[0098] Input module 11, used to input source image into known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image;

[0099] A matching module 12 is used to extract each image frame of the video and match each extracted image frame with the camera posture information;

[0100] The recognition module 13 is used to recognize the source image and obtain a text description related to the source image;

[0101] The generation module 14 is used to use the text description related to the source image and each matching image frame and camera pose information as input to a known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image.

[0102] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0103] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0104] In the above embodiment, any multiple of the input module 11, the matching module 12, the identification module 13 and the generation module 14 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. At least one of the input module 11, the matching module 12, the identification module 13 and the generation module 14 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the input module 11, the matching module 12, the identification module 13 and the generation module 14 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0105] See also Figure 6 , an electronic device provided by an embodiment of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0106] Memory 1130, used for storing computer programs;

[0107] The processor 1110 is used to implement the following three-dimensional reconstruction method when executing the program stored in the memory 1130:

[0108] Analyze satellite status information, positioning error information and satellite navigation time from the received GPS message information;

[0109] The source image is fed into a known multi-view Figure 3 dimensional reconstruction network, outputting video and camera pose information corresponding to the source image;

[0110] Extracting each image frame of the video, and matching each extracted image frame with camera pose information;

[0111] Identify the source image and obtain text description related to the source image;

[0112] The text description related to the source image and each matching image frame and camera pose information are used as input to a known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image.

[0113] The communication bus 1140 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0114] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0115] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory 1130 may also be at least one storage device located away from the processor 1110.

[0116] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0117] The embodiment of the present disclosure further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional reconstruction method described above is implemented.

[0118] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the three-dimensional reconstruction method according to the embodiment of the present disclosure is implemented.

[0119] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0120] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0121] The foregoing is merely a specific embodiment of the present disclosure, which enables those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Input the source image into a known multi-view 3D reconstruction network and output the video and camera pose information corresponding to the source image; Extracting each image frame of the video, and matching each extracted image frame with camera pose information; Identify the source image and obtain text description related to the source image; The text description related to the source image and the matching information of each image frame and camera pose are used as the input of the known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image. The image generation model includes a diffusion model and a three-dimensional reconstruction unit. The text description related to the source image and each image frame and camera pose information that match each other are used as inputs of the known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image, including: Input each image frame and camera pose information that matches each other into a 3D reconstruction unit to obtain an initial 3D reconstruction unit; Inputting a text description related to a source image into a known diffusion model to generate a first image corresponding to the source image; Generate a rendering using an initial 3D reconstruction unit; Training an initial 3D reconstruction unit according to the first image and the rendering to obtain a known 3D reconstruction unit; Based on the known three-dimensional reconstruction unit, a three-dimensional model corresponding to the source image is obtained.

2. The method according to claim 1, characterized in that The method of inputting a source image into a known multi-view 3D reconstruction network and outputting a video and camera pose information corresponding to the source image comprises: The source image is input into a known multi-view 3D reconstruction network to generate a target 3D model corresponding to the source image; Collect images at different angles around the target 3D model and calculate the camera pose at each acquisition angle; A video corresponding to the source image is generated according to an image sequence formed by images at different angles.

3. The method according to claim 1, characterized in that: The identifying of the source image to obtain a text description related to the source image includes: The pre-trained image recognition model is used to identify the source image and obtain text descriptions related to the source image.

4. The method according to claim 1, characterized in that: The step of inputting the text description related to the source image into a known diffusion model to generate a first image corresponding to the source image includes: Mapping text descriptions associated with source images into embedding vectors for generated images; A first image corresponding to the source image is generated according to the embedding vector.

5. The method according to claim 1, characterized in that The step of training an initial 3D reconstruction unit according to the first image and the rendering image to obtain a known 3D reconstruction unit includes: Using a known diffusion model, gradually adding noise to the rendering image to obtain a first Gaussian noise corresponding to the rendering image; Denoising the first Gaussian noise using the diffusion model used to generate the first image to obtain a second image; The noise between the second image and the first image is used as the second Gaussian noise; Calculate the noise loss according to the rendering image, the first Gaussian noise and the second image; When the noise loss is less than a preset threshold, the initial 3D reconstruction unit is used as a known 3D reconstruction unit; When the noise loss is greater than or equal to the preset threshold, the parameters of the initial 3D reconstruction unit are adjusted, and the noise loss is recalculated for the 3D reconstruction unit after the parameter adjustment until the noise loss is less than the preset threshold.

6. The method according to claim 5, characterized in that The recalculating the noise loss for the three-dimensional reconstruction unit after the parameter adjustment includes: Returning to the step of generating a first image corresponding to the source image; In the case that the input of the diffusion model is a text description when the first image was generated last time, the first image is generated by using the image as the input of the known diffusion model, wherein the image is selected from a set including a source image and an input 3D reconstruction unit image frame; When the input of the diffusion model is an image when the first image is generated last time, the first image is generated using the text description as the input of the known diffusion model.

7. A three-dimensional reconstruction device, characterized in that: include: An input module, used for inputting a source image into a known multi-view 3D reconstruction network, and outputting a video and camera pose information corresponding to the source image; A matching module, used for extracting each image frame of the video and matching each extracted image frame with camera pose information; A recognition module, used to recognize the source image and obtain a text description related to the source image; The generation module is used to use the text description related to the source image and each image frame and camera pose information that match each other as inputs to the known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image. The image generation model includes a diffusion model and a three-dimensional reconstruction unit. The text description related to the source image and each image frame and camera pose information that match each other are used as inputs of the known image generation model to generate images of various angles corresponding to the source image to form a three-dimensional model corresponding to the source image, including: Input each image frame and camera pose information that matches each other into a 3D reconstruction unit to obtain an initial 3D reconstruction unit; Inputting a text description related to a source image into a known diffusion model to generate a first image corresponding to the source image; Generate a rendering using an initial 3D reconstruction unit; Training an initial 3D reconstruction unit according to the first image and the rendering to obtain a known 3D reconstruction unit; Based on the known three-dimensional reconstruction unit, a three-dimensional model corresponding to the source image is obtained.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the three-dimensional reconstruction method according to any one of claims 1 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 6 is implemented.

Images