Three-dimensional reconstruction model training, image reconstruction method and device

Abstract

The present disclosure provides a three-dimensional reconstruction model training, image reconstruction method and device, which relates to the field of artificial intelligence technology, specifically computer vision, augmented reality, virtual reality, deep learning and other technical fields, and can be applied to artificial intelligence content generation, virtual image and other scenes. The specific implementation scheme is: obtain a preset pose sample set, determine a first loss function based on preset noise and estimated noise, and train the initial reconstruction model based on the first loss function to obtain a reconstruction model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, augmented reality, virtual reality, deep learning and other technical fields, and can be applied to scenes such as artificial intelligence content generation and virtual images, and in particular to a three-dimensional reconstruction model training, image reconstruction method and device. Background Art

[0002] Three-dimensional reconstruction is a technology that constructs three-dimensional data from pre-processed two-dimensional image sequences and displays it on a computer screen.

[0003] Existing technologies usually use NeuS (Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction) and SFM (Structure From Motion) to reconstruct three-dimensional images of objects. Summary of the invention

[0004] The embodiments of the present disclosure provide a three-dimensional reconstruction model training, image reconstruction method, device, equipment and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction model training method, the method comprising: obtaining a preset set of pose samples; determining a first loss function based on preset noise and estimated noise; and training an initial reconstruction model based on the first loss function to obtain a reconstructed model.

[0006] In a second aspect, an embodiment of the present disclosure provides an image reconstruction method, the method comprising: generating a new pose image of an object to be reconstructed based on a reconstruction model, wherein the reconstruction model is a reconstruction model obtained by the method described in any implementation manner of the first aspect above; based on the new pose image, reconstructing a three-dimensional image of the object to be reconstructed.

[0007] In a third aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction model training device, which includes: a sample acquisition module, configured to acquire a preset set of pose samples; a loss determination module, configured to determine a first loss function based on preset noise and estimated noise; and a first training module, configured to train an initial reconstruction model based on the first loss function to obtain a reconstructed model.

[0008] In a fourth aspect, an embodiment of the present disclosure provides an image reconstruction device, comprising: an image generation module, configured to generate a new pose image of an object to be reconstructed based on a reconstruction model, wherein the reconstruction model is obtained by the method described in any implementation manner of the first aspect above; and an image reconstruction module, configured to reconstruct a three-dimensional image of the object to be reconstructed based on the new pose image.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement a three-dimensional reconstruction model training method as in any embodiment of the first aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect or the second aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any embodiment of the first aspect or the second aspect.

[0012] The present disclosure effectively improves the accuracy of the reconstruction model in generating images of new perspectives, thereby improving the accuracy of three-dimensional images reconstructed based on the trained reconstruction model.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0015] Figure 2 is a flowchart of an embodiment of a three-dimensional reconstruction model training method according to the present disclosure;

[0016] Figure 3 is a schematic diagram of an application scenario of the three-dimensional reconstruction model training method disclosed in the present invention;

[0017] Figure 4 is a flowchart of another embodiment of the image reconstruction method according to the present disclosure;

[0018] Figure 5 is a schematic diagram of an embodiment of a three-dimensional reconstruction model training device according to the present disclosure;

[0019] Figure 6 is a schematic diagram of an embodiment of an image reconstruction device according to the present disclosure;

[0020] Figure 7 It is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the 3D reconstruction model training method of the present disclosure can be applied.

[0024] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0025] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0026] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with image acquisition functions, including but not limited to cameras and webcams. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide three-dimensional reconstruction model training services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] The server 105 may be a server that provides various services, for example, obtaining a preset set of pose samples; determining a first loss function based on preset noise and estimated noise; and training an initial reconstruction model based on the first loss function to obtain a reconstruction model.

[0028] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide 3D reconstruction model training services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0029] It should be noted that the 3D reconstruction model training method provided in the embodiments of the present disclosure can be executed by the server 105, or by the terminal devices 101, 102, 103, or by the server 105 and the terminal devices 101, 102, 103 cooperating with each other. Accordingly, the various parts (such as various units, sub-units, modules, and sub-modules) included in the 3D reconstruction model training device can be all set in the server 105, or all set in the terminal devices 101, 102, 103, or respectively set in the server 105 and the terminal devices 101, 102, 103.

[0030] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0031] Figure 2 A process 200 of an embodiment of a 3D reconstruction model training method is shown. The 3D reconstruction model training method comprises the following steps:

[0032] Step 201, obtaining a preset pose sample set.

[0033] In this embodiment, the execution subject (for example, Figure 1 The server 105 or terminal devices 101, 102, 103) can obtain a preset pose sample set in a wired or wireless manner.

[0034] Among them, the preset posture sample set may include one or more first posture samples, and the posture information indicated by each first posture sample is different. The posture information is usually used to indicate the position and orientation of the image acquisition device when performing image acquisition.

[0035] The above wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (Ultra Wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0036] Step 202: determine a first loss function based on the preset noise and the estimated noise.

[0037] In this embodiment, the execution subject may first input the first pose sample into the initial reconstruction model to obtain the first pose image including the object to be reconstructed.

[0038] The object to be reconstructed may be any object to be reconstructed, such as a doll, a water cup, etc., and this application does not limit this.

[0039] Here, the initial reconstruction model can be a model in existing technology or future development technology that can perform three-dimensional reconstruction based on the input posture, such as the NeRF (Neural Radiance Field) model, the SDF (Signed Distance Field) model, the DMTet (Deep Marching Tetrahedra) model, etc., which is not limited in this application.

[0040] Furthermore, the estimated noise is obtained in the following manner: a specified reference image of the object to be reconstructed, the position and posture of the specified reference image, a first pose image, a first pose sample, and a preset noise are input into a preset diffusion model to obtain the estimated noise.

[0041] The preset noise may be a noise with a stable distribution in the existing technology or future development technology, such as Gaussian noise, Gaussian white noise, etc., and this application does not limit this.

[0042] Here, the pose of the specified reference image is usually different from the first pose sample.

[0043] Among them, the diffusion model can be a diffusion model for generating a target image based on a reference image, a reference image posture, and a target image posture in existing technology or future development technology, for example, a Zero123 (View-Conditioned 2DDiffusion, two-dimensional diffusion under viewing angle (posture) conditions) model.

[0044] Here, the Zero123 model is a new perspective image generation framework based on diffusion models. For the Zero123 model, you only need to input a single image containing a specified object and select another camera perspective to generate a two-dimensional image of the specified object under the other camera perspective.

[0045] The Zero123 model may generally include a U-net network, wherein the U-net network is used to denoise the noisy image based on an input reference image, a pose of the reference image, and a pose of the target image.

[0046] Here, the diffusion model can also generate estimated noise based on the U-net network to evaluate whether a specified image refers to a reasonable image in a positioning pose, that is, the degree of match between the specified image and the specified pose, that is, the degree of similarity between the specified image and the real image in the specified pose.

[0047] Furthermore, a first loss function is determined according to a difference between the preset noise and the estimated noise.

[0048] Here, the difference between the preset noise and the estimated noise can be used to evaluate the matching degree between the first pose image and the first pose sample. The difference is negatively correlated with the matching degree. The smaller the difference, the higher the matching degree, and the larger the difference, the smaller the matching degree.

[0049] Specifically, the first loss function It can be expressed by the following formula:

[0050]

[0051] Among them, ∈ is Gaussian white noise, To estimate the noise.

[0052] Step 203: Based on the first loss function, the initial reconstruction model is trained to obtain a reconstruction model.

[0053] In this embodiment, after determining the first loss function, the execution subject may train the initial reconstruction model according to the first loss function and the constraints of the reconstruction model itself (such as cross entropy loss, etc.) to obtain the reconstruction model.

[0054] In some optional embodiments, the method further includes: training the reconstruction model based on at least one second pose sample and a second pose image corresponding to each second pose sample to obtain a new reconstruction model.

[0055] In this implementation, the preset pose sample set may also include one or more second pose samples, the pose information indicated by each second pose sample is different, each second pose sample corresponds to a second pose image, and the second pose image is a real image including the object to be reconstructed and having the pose information indicated by the second pose sample.

[0056] The executing entity can directly construct the target loss function based on the predicted image obtained by inputting the second pose sample into the reconstruction model, and the second pose image corresponding to the second pose sample of the input model, or it can construct the target loss function based on the three-dimensional estimation information of the predicted image obtained by inputting the second pose sample into the reconstruction model, and the real three-dimensional estimation information of the second pose image corresponding to the second pose sample of the input model. This application does not limit this.

[0057] Furthermore, the reconstruction model is trained based on the target loss function to obtain a new reconstruction model.

[0058] This implementation method obtains a new reconstruction model by training the reconstruction model based on at least one second pose sample and the second pose image corresponding to each second pose sample, thereby further improving the accuracy of the reconstruction model obtained through training.

[0059] In some optional methods, a reconstruction model is trained based on at least one second pose sample and a second pose image corresponding to each second pose sample to obtain a new reconstruction model, including: determining a second loss function based on real three-dimensional estimation information of the second pose image corresponding to the second pose sample and target three-dimensional estimation information of the predicted image; and training the reconstruction model based on the second loss function to obtain a new reconstruction model.

[0060] In this implementation, the preset pose sample set may also include one or more second pose samples, and each second pose sample in at least one second pose sample corresponds to real three-dimensional estimation information of the second pose image, and the three-dimensional estimation information may include a mask, a depth map, a normal map, etc.

[0061] A second loss function is determined based on the real three-dimensional estimation information of the second pose image corresponding to the second pose sample input into the reconstruction model and the target three-dimensional estimation information of the predicted image obtained by inputting the second pose sample into the reconstruction model.

[0062] Specifically, the second loss function It can be expressed by the following formula:

[0063]

[0064] Among them, M′, D′ are respectively the mask and depth map in the target 3D estimation information of the predicted image, and M, D are respectively the mask and depth map in the real 3D estimation information of the second pose image corresponding to the second pose sample.

[0065] Furthermore, the reconstruction model is trained based on the second loss function to obtain a new reconstruction model.

[0066] This implementation method determines a second loss function through the real three-dimensional estimation information of the second pose image corresponding to the second pose sample and the target three-dimensional estimation information of the predicted image; trains the reconstruction model based on the second loss function to obtain a new reconstruction model, thereby further improving the accuracy of the trained reconstruction model.

[0067] In some optional embodiments, the three-dimensional estimation information includes at least one of the following: a mask, a depth map, and a normal map.

[0068] In this implementation, the execution entity can determine the second loss function based on the real three-dimensional estimation information of the second pose image corresponding to the second pose sample of the input reconstruction model and the target three-dimensional estimation information of the predicted image.

[0069] The three-dimensional estimation information may include at least one of the following: a mask, a depth map, and a normal map.

[0070] Specifically, the three-dimensional estimation information may include: mask, depth map, normal map, and the second loss function It can be expressed by the following formula:

[0071]

[0072] Among them, M′, D′, N′ are the mask, depth map and normal map in the target 3D estimation information of the predicted image, respectively, and M, D, N are the mask, depth map and normal map in the real 3D estimation information of the second pose image, respectively.

[0073] This implementation takes into account 3D estimation information of different dimensions, which helps to further improve the accuracy of the trained reconstruction model.

[0074] In some optional methods, a reconstruction model is trained based on at least one second pose sample and a second pose image corresponding to each second pose sample to obtain a new reconstruction model, including: determining a third loss function based on the second pose sample, the real three-dimensional estimation information of the second pose image corresponding to the second pose sample, and the pose of the predicted image, and the target three-dimensional estimation information of the predicted image; and training the reconstruction model based on the third loss function to obtain a new reconstruction model.

[0075] In this implementation, the execution entity can determine a third loss function based on the second pose sample, the real three-dimensional estimation information of the second pose image corresponding to the second pose sample input into the reconstruction model, the pose of the predicted image obtained by inputting the second pose sample into the reconstruction model, and the target three-dimensional estimation information of the predicted image; and continue to train the reconstruction model based on the third loss function to obtain a new reconstruction model.

[0076] Specifically, the third loss function can be expressed by the following formula:

[0077]

[0078] Among them, I′ is the pose of the predicted image, M′, D′, N′ are the mask, depth map and normal map in the target 3D estimation information of the predicted image, I is the second pose sample, M, D, N are the mask, depth map and normal map in the real 3D estimation information corresponding to the second pose sample, respectively.

[0079] This implementation method determines a third loss function based on the second pose sample, the real three-dimensional estimation information of the second pose image corresponding to the second pose sample, the pose of the predicted image, and the target three-dimensional estimation information of the predicted image; trains the reconstruction model based on the third loss function to obtain a new reconstruction model, thereby further improving the accuracy of the trained reconstruction model.

[0080] In some optional embodiments, the initial reconstructed model is a neural radiation field model.

[0081] In this implementation, the initial reconstruction model is a neural radiation field model, namely, a NeRF model.

[0082] Here, NeRF is used to synthesize images of the same scene from different perspectives, i.e., poses. The images from different perspectives can be further used to reconstruct a 3D representation of the scene.

[0083] NeRF includes several fully connected networks. Specifically, the training process of NeRF may include: the camera position can be encoded and input into the fully connected network to obtain density and color, that is, the particle density and color value corresponding to a sampling point on a ray, where the density is used to calculate the weight to obtain the presented pixel color by weighted summation of the color at the point. Further, volume rendering is performed based on the particle density and color of all sampling points on the ray to obtain a predicted two-dimensional image, that is, a projection view. Finally, the loss function is calculated based on the projection view and the real projection view corresponding to the camera position to train the network.

[0084] This implementation method helps to improve the quality of three-dimensional images reconstructed by the trained reconstruction model by adopting the neural radiation field model as the initial reconstruction model.

[0085] Continue to see Figure 3 , Figure 3 It is a schematic diagram of an application scenario of the three-dimensional reconstruction model training method according to this embodiment. The execution subject 301 can obtain a preset pose sample set 302 in a wired or wireless manner. The pose sample set 302 may include multiple first pose samples, and the pose information indicated by each first pose sample is different. The pose information is usually used to indicate the position and orientation of the image acquisition device when performing image acquisition, for example, the shooting angle of the camera. The first pose sample is input into the initial reconstruction model 303 (for example, NeRF model) to obtain the first pose image including the object to be reconstructed. Further, the estimated noise is obtained in the following manner: the specified reference image including the object to be reconstructed (for example, a doll), the pose of the specified reference image, the first pose image, the first pose sample, and the preset noise are input into the preset diffusion model (for example, Zero123) 304 to obtain the estimated noise. According to the difference between the preset noise and the estimated noise, the first loss function is determined, and based on the first loss function, the initial reconstruction model 303 is trained to obtain the trained reconstruction model.

[0086] The three-dimensional reconstruction model training method provided by the embodiments of the present disclosure obtains a preset set of pose samples, determines a first loss function based on preset noise and estimated noise, and trains an initial reconstruction model based on the first loss function to obtain a reconstruction model. That is, on the basis of fewer images used to train the reconstruction model, the rationality of the predicted image of the new perspective generated by the initial reconstruction model is judged by a diffusion model, and the initial reconstruction model is trained based on the judgment result, which effectively improves the accuracy of the image of the new perspective generated by the reconstruction model, and then improves the accuracy of the three-dimensional image reconstructed based on the trained reconstruction model.

[0087] Further references Figure 4 , which shows a process 400 of an embodiment of an image reconstruction method. In this embodiment, the process 400 of the image reconstruction method may include the following steps:

[0088] Step 401: Generate a new pose image of the object to be reconstructed based on the reconstruction model.

[0089] In this embodiment, the execution subject (for example, Figure 1 The server 105 or terminal devices 101, 102, 103) can use the reconstruction model stored in the local or cloud server to generate a new pose image of the object to be reconstructed, that is, a new perspective image.

[0090] The reconstruction model is as follows Figure 2 The reconstructed model obtained by the method described in the corresponding embodiment will not be described in detail here.

[0091] Step 402: reconstruct a three-dimensional image of the object to be reconstructed based on the new pose image.

[0092] In this embodiment, after obtaining the new pose image, the execution subject may render the new pose image to reconstruct a three-dimensional image of the object to be reconstructed.

[0093] Specifically, the reconstruction model is a NeRF model. After generating a new perspective image based on the NeRF model, the voxel density isosurface can be extracted from the trained NeRF model using a marching cube method to obtain a mesh of the 3D object.

[0094] The image reconstruction method provided by the embodiment of the present disclosure generates a new pose image of the object to be reconstructed based on a reconstruction model, and reconstructs a three-dimensional image of the object to be reconstructed based on the new pose image, thereby effectively improving the precision and accuracy of the reconstructed three-dimensional image.

[0095] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a 3D reconstruction model training device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0096] like Figure 5 As shown, the 3D reconstruction model training device 500 of this embodiment includes: a sample acquisition module 501 , a loss determination module 502 and a first training module 503 .

[0097] The sample acquisition module 501 may be configured to acquire a preset pose sample set.

[0098] The loss determination module 502 may be configured to determine a first loss function based on the preset noise and the estimated noise.

[0099] The first training module 503 may be configured to train the initial reconstruction model based on the first loss function to obtain a reconstruction model.

[0100] In some optional embodiments of this embodiment, the device also includes: a second training module, configured to train the reconstruction model based on multiple second posture samples and second posture images corresponding to each second posture sample to obtain a new reconstruction model.

[0101] In some optional embodiments of this embodiment, the second training module is further configured to: determine a second loss function based on the real three-dimensional estimation information of the second pose image corresponding to the second pose sample, and the target three-dimensional estimation information of the predicted image; train the reconstruction model based on the second loss function to obtain a new reconstruction model.

[0102] In some optional embodiments of this embodiment, the second training module is further configured to: determine a third loss function based on the second pose sample, the real three-dimensional estimation information of the second pose image corresponding to the second pose sample, and the pose of the predicted image, and the target three-dimensional estimation information of the predicted image; train the reconstruction model based on the third loss function to obtain a new reconstruction model.

[0103] In some optional aspects of this embodiment, the initial reconstruction model is a neural radiation field model.

[0104] In some optional aspects of this embodiment, the three-dimensional estimation information includes at least one of the following: a mask, a depth map, and a normal map.

[0105] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0106] Further references Figure 6 , the present application provides an embodiment of an image reconstruction device. The device embodiment is Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 6 As shown, the image reconstruction device 600 of this embodiment includes: an image generating module 601 and an image reconstructing module 602 .

[0108] The image generation module 601 may be configured to generate a new pose image of the object to be reconstructed based on the reconstruction model.

[0109] The image reconstruction module 602 may be configured to reconstruct a three-dimensional image of the object to be reconstructed based on the new posture image.

[0110] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0111] like Figure 7 , is a block diagram of an electronic device according to a three-dimensional reconstruction model training method according to an embodiment of the present disclosure.

[0112] 700 is a block diagram of an electronic device according to a three-dimensional reconstruction model training method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0113] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0114] The memory 702 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the three-dimensional reconstruction model training method provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the three-dimensional reconstruction model training method provided by the present disclosure.

[0115] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the 3D reconstruction model training method in the embodiment of the present disclosure (for example, the attached Figure 5 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the 3D reconstruction model training method in the above method embodiment is implemented.

[0116] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device for face tracking, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely disposed relative to the processor 701, and these remote memories may be connected to the electronic device for lane line detection via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0117] The electronic device of the 3D reconstruction model training method may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0118] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for lane line detection, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0119] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0123] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0124] According to the technical solution of the embodiment of the present disclosure, the accuracy of the reconstruction model in generating images of new perspectives is effectively improved, thereby improving the accuracy of the three-dimensional images reconstructed based on the trained reconstruction model.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and this document is not limited here.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A three-dimensional reconstruction model training method, comprising: Acquire a preset pose sample set, the pose sample set comprising: at least one first pose sample and at least one second pose sample, for each second pose sample, the second pose sample corresponds to a second pose image, the second pose image is a real image including the object to be reconstructed and having the pose information indicated by the second pose sample; Determine a first loss function based on preset noise and estimated noise, wherein the estimated noise is obtained by: inputting a specified reference image of an object to be reconstructed, a pose of the specified reference image, a first pose image, a first pose sample, and preset noise into a preset diffusion model to obtain the estimated noise, and the first pose image is obtained by inputting the first pose sample into an initial reconstruction model; Based on the first loss function, the initial reconstruction model is trained to obtain a reconstruction model; Based on at least one second pose sample and the second pose images corresponding to each second pose sample, the reconstruction model is trained to obtain a new reconstruction model.

2. The method according to claim 1, wherein: The method of training the reconstruction model based on at least one second posture sample and the second posture images corresponding to each second posture sample to obtain a new reconstruction model includes: determining a second loss function based on true three-dimensional estimation information of a second pose image corresponding to the second pose sample and target three-dimensional estimation information of a predicted image obtained by inputting the second pose sample into the reconstruction model; Based on the second loss function, the reconstruction model is trained to obtain a new reconstruction model.

3. The method according to claim 1, wherein: The method of training the reconstruction model based on at least one second posture sample and the second posture images corresponding to each second posture sample to obtain a new reconstruction model includes: determining a third loss function based on the second pose sample, the real three-dimensional estimation information of the second pose image corresponding to the second pose sample, the pose of the predicted image, and the target three-dimensional estimation information of the predicted image, wherein the predicted image is obtained by inputting the second pose sample into the reconstruction model; Based on the third loss function, the reconstruction model is trained to obtain a new reconstruction model.

4. The method according to claim 2, wherein: The three-dimensional estimation information includes at least one of the following: a mask, a depth map, and a normal map.

5. The method according to any one of claims 1 to 4, wherein: The initial reconstruction model is a neural radiation field model.

6. An image reconstruction method, comprising: Generate a new pose image of the object to be reconstructed based on the reconstruction model, wherein the reconstruction model is a reconstruction model obtained by the method according to any one of claims 1 to 5; Based on the new pose image, a three-dimensional image of the object to be reconstructed is reconstructed.

7. A three-dimensional reconstruction model training device, comprising: A sample acquisition module is configured to acquire a preset pose sample set, wherein the pose sample set includes: at least one first pose sample and at least one second pose sample, and for each second pose sample, the second pose sample corresponds to a second pose image, and the second pose image is a real image including the object to be reconstructed and having the pose information indicated by the second pose sample; A loss determination module is configured to determine a first loss function based on preset noise and estimated noise, wherein the estimated noise is obtained by: inputting a specified reference image including an object to be reconstructed, a position and posture of the specified reference image, a first position image, a first position sample, and the preset noise into a preset diffusion model to obtain the estimated noise, and the first position image is obtained by inputting the first position sample into an initial reconstruction model; A first training module is configured to train the initial reconstruction model based on the first loss function to obtain a reconstruction model; The second training module is configured to train the reconstruction model based on at least one second posture sample and the second posture image corresponding to each second posture sample to obtain a new reconstruction model.

8. The device according to claim 7, wherein: The second training module is further configured to: determining a second loss function based on true three-dimensional estimation information of a second pose image corresponding to the second pose sample and target three-dimensional estimation information of a predicted image obtained by inputting the second pose sample into the reconstruction model; Based on the second loss function, the reconstruction model is trained to obtain a new reconstruction model.

9. The device according to claim 7, wherein: The second training module is further configured to: determining a third loss function based on the second pose sample, the real three-dimensional estimation information of the second pose image corresponding to the second pose sample, the pose of the predicted image, and the target three-dimensional estimation information of the predicted image, wherein the predicted image is obtained by inputting the second pose sample into the reconstruction model; Based on the third loss function, the reconstruction model is trained to obtain a new reconstruction model.

10. The device according to claim 8, wherein: The three-dimensional estimation information includes at least one of the following: a mask, a depth map, and a normal map.

11. The device according to any one of claims 7 to 10, wherein: The initial reconstruction model is a neural radiation field model.

12. An image reconstruction device, comprising: An image generation module is configured to generate a new pose image of the object to be reconstructed based on the reconstruction model, wherein the reconstruction model is a reconstruction model obtained by the method according to any one of claims 1 to 5; The image reconstruction module is configured to reconstruct a three-dimensional image of the object to be reconstructed based on the new posture image.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images