Three-dimensional model reconstruction and image generation method, device, storage medium, and program product

Abstract

Embodiments of the present application provide a three-dimensional model reconstruction and image generation method, a device, a storage medium, and a program product. In the method, multi-stage three-dimensional reconstruction is performed on the basis of a single image of a target object; in a first stage, a plurality of view images are generated on the basis of an image generation model, and an initial three-dimensional model is reconstructed on the basis of the plurality of view images; and in a second stage, on the basis of the plurality of view images and an initial prompt containing set marker information, a text-to-image model is used to learn an association relationship between the target object and the set marker information, and a plurality of scene images are generated on this basis. Compared with the plurality of view images, the scene images generated in the second stage have higher resolution and richer image details, and then the initial three-dimensional model is optimized on the basis of the plurality of scene images, so that a target three-dimensional model having higher resolution and clearer model details can be obtained, thereby paving the way for practical application of a three-dimensional reconstruction solution based on a single image.

Description

Three-dimensional model reconstruction and image generation method and device, storage medium and program product

[0001] Cross-reference to related applications

[0002] This application is based on the Chinese Patent Application No. 202410713099.3 entitled "Three-dimensional model reconstruction and image generation method and device, storage medium and program product" filed on June 3, 2024, which is incorporated by reference in its entirety into this application. TECHNICAL FIELD

[0003] The present application relates to the technical field of computer vision, and in particular to a three-dimensional model reconstruction and image generation method, device, storage medium and program product. BACKGROUND

[0004] Image-based three-dimensional (3D) reconstruction refers to a technology of reconstructing a three-dimensional model, such as a point cloud model or a mesh model, of an object from a two-dimensional image. With the development of three-dimensional application scenarios and the emergence of hardware devices supporting 3D applications, the demand for 3D models is increasing, and at the same time, the requirement for the detail expression capability of 3D models is also increasing.

[0005] At present, the three-dimensional reconstruction based on multiple images is relatively mature, that is, multiple images of an object are captured by a camera, consistent feature points in the multiple images are matched, and parameters of the camera are estimated from the multiple images, and the three-dimensional reconstruction of the object is performed by using the feature points and the camera parameters. However, with the continuous development of computer vision technology, the industry has begun to challenge the three-dimensional reconstruction based on a single image.

[0006] However, a single image contains limited information and is also affected by various factors such as cameras, lighting, and object occlusion relationships, resulting in that the three-dimensional reconstruction based on a single image is mainly limited to objects with more prior information, such as human faces. Even so, the three-dimensional reconstruction based on a single image is still not very ideal, and the 3D model reconstructed has quality problems such as low resolution and unclear model details, and it is difficult to directly apply it. SUMMARY

[0007] Aspects of the present application provide a three-dimensional model reconstruction and image generation method, device, storage medium and program product to improve the resolution and model detail clarity of a three-dimensional model reconstructed based on a single image.

[0008] The embodiment of the present application provides a three-dimensional model reconstruction and image generation method, device, storage medium and program product, which comprises the following steps: generating a plurality of perspective images corresponding to a target object based on an original image of the target object and a plurality of perspective information by using a target image generation model; performing three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object; learning an association relationship between the target object and setting mark information according to the plurality of perspective images and an initial prompt word containing the setting mark information by using a text-to-image model; generating a plurality of scene images corresponding to the target object by using the association relationship between the target object and the setting mark information learned by the text-to-image model according to a plurality of scene prompt words containing the setting mark information; and optimizing the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object.

[0009] The embodiment of the present application also provides an image generation method, which comprises the following steps: generating a plurality of perspective images corresponding to a target object based on an original image of the target object and a plurality of perspective information by using a target image generation model; performing three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object; learning an association relationship between the target object and setting mark information according to the plurality of perspective images and an initial prompt word containing the setting mark information by using a text-to-image model; and generating a plurality of scene images corresponding to the target object by using the association relationship between the target object and the setting mark information learned by the text-to-image model according to a plurality of scene prompt words containing the setting mark information.

[0010] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor; the memory is used for storing a computer program; the processor is coupled with the memory and is used for executing the computer program in the memory to implement the steps in the three-dimensional model reconstruction method or the image generation method provided by the embodiment of the present application.

[0011] The embodiment of the present application also provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can implement the steps in the three-dimensional model reconstruction method or the image generation method provided by the embodiment of the present application.

[0012] The embodiment of the present application also provides a computer program product, which comprises a computer program / instruction, when the computer program / instruction is executed by a processor, the processor can implement the steps in the three-dimensional model reconstruction method and the image generation method.

[0013] In the embodiments of the present application, multi-stage three-dimensional reconstruction is performed based on a single image of a target object. In the first stage, a plurality of perspective images are generated based on an image generation model, and an initial three-dimensional model is reconstructed based on the generated plurality of perspective images. In the second stage, based on the plurality of perspective images and an initial prompt word containing setting mark information, an association between the target object and the setting mark information is learned by using a text-to-image model, and on this basis, a plurality of scene images are generated. Compared with the plurality of perspective images, the scene images generated in the second stage have higher resolution and more detailed image details, and then the initial three-dimensional model is optimized based on the plurality of scene images, so that a target three-dimensional model with higher resolution and clearer model details can be obtained, thereby providing conditions for the landing application of the single-image-based three-dimensional reconstruction scheme. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate embodiments of the present application and the description thereof, and do not constitute improper limitations on the present application. In the drawings:

[0015] FIG. 1a is a flowchart of a three-dimensional model reconstruction method provided by an example embodiment of the present application;

[0016] FIG. 1b is a model architecture diagram of the three-dimensional model reconstruction method provided by an example embodiment of the present application;

[0017] FIG. 2a is a structural diagram of a target image generation model provided by an example embodiment of the present application;

[0018] FIG. 2b is a structural diagram of a target image generation model provided by another example embodiment of the present application;

[0019] FIG. 3 is a structural diagram of a text-to-image model provided by an example embodiment of the present application;

[0020] FIG. 4 is a structural diagram of a text-to-image model provided by another example embodiment of the present application;

[0021] FIG. 5 is a flowchart of an image generation method provided by an example embodiment of the present application;

[0022] FIG. 6 is a structural diagram of a three-dimensional model reconstruction and image generation apparatus provided by an example embodiment of the present application;

[0023] FIG. 7 is a structural diagram of an electronic device provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0025] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to choose authorization or refusal. In addition, the various models (including but not limited to language models or large models) involved in the present application are in line with relevant legal and standard regulations.

[0026] In the prior art, the three-dimensional reconstruction result based on a single image is not very ideal, and the 3D model reconstructed generally has problems such as low resolution and unclear model details, and it is difficult to directly apply it.

[0027] In view of the above technical problems, the embodiments of the present application provide a solution, which can perform multi-stage three-dimensional reconstruction based on a single image of a target object. In the first stage, a plurality of perspective images are generated based on an image generation model, and an initial three-dimensional model is reconstructed based on the plurality of perspective images. In the second stage, based on the plurality of perspective images and an initial prompt word containing set marker information, an association between the target object and the set marker information is learned by using a text-to-image model, and on this basis, a plurality of scene images are generated. Compared with the plurality of perspective images, the scene images generated in the second stage have higher resolution and more image details, and then the initial three-dimensional model is optimized based on the plurality of scene images, so that a target three-dimensional model with higher resolution and clearer model details is obtained, which provides conditions for the landing application of the three-dimensional reconstruction scheme based on a single image.

[0028] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0029] FIG. 1a is a flowchart of a three-dimensional model reconstruction method according to an example embodiment of the present application, FIG. 1b is a model architecture diagram of the three-dimensional model reconstruction method according to an embodiment of the present application, and the numbers 11, 12, 13, 14 and 15 in the circles in FIG. 1b correspond to steps 11, 12, 13, 14 and 15 respectively. The dashed box in FIG. 1b is used to indicate that the detail repair of the scene image based on the image inpainting model is an optional operation. For this part, see the subsequent embodiments. The method can be executed by an electronic device, which can be a terminal device such as a smartphone, a tablet computer, a computer or a smartwatch, or a server device such as a conventional server, a cloud server or a server cluster, and the present embodiment does not make any limitation. As shown in FIG. 1a and FIG. 1b, the method can include the following steps:

[0030] Step 11, generating a plurality of perspective images corresponding to the target object based on the original image of the target object and a plurality of perspective information, using a target image generation model.

[0031] Step 12, performing three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object.

[0032] Step 13, learning the association between the target object and the set marker information according to the plurality of perspective images and the initial prompt word containing the set marker information, using a text-to-image model.

[0033] Step 14, generating a plurality of scene images corresponding to the target object based on the association between the target object and the set marker information learned by the text-to-image model, according to a plurality of scene prompt words containing the set marker information.

[0034] Step 15, optimizing the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object.

[0035] In the present embodiment, the target object can be any object, such as an animal, a plant, a building, a person, etc., or various goods that can be sold on an e-commerce platform, such as a vehicle, a mobile phone, a table, a sofa, clothes, sports goods, etc., and the present embodiment does not make any limitation.

[0036] In this embodiment, the target object is reconstructed in three dimensions based on a single original image of the target object, that is, only one image of the target object needs to be captured or collected. In this embodiment, the image is referred to as an original image, and the angle of view information when the original image is captured is not limited. The electronic device responsible for three-dimensional reconstruction can obtain the original image of the target object. Alternatively, the electronic device can have a camera, which is used to capture the target object and thus obtain the original image of the target object; or an independent camera device can be used to capture the target object and upload the captured original image of the target object to the electronic device.

[0037] In this embodiment, in order to better reconstruct the target object in three dimensions based on a single original image, multiple angle of view information can be obtained, and then images of the target object under multiple angle of view information are generated based on the original image, and the image of the target object under any angle of view information is referred to as an angle of view image. The angle of view information in this embodiment is used to describe the angle of view for observing the target object. The multiple angle of view information can be set by the user, or randomly set by the electronic device when the three-dimensional model is reconstructed, or pre-set. This embodiment is not limited.

[0038] In this embodiment, the target image generation model M1 having a multi-angle image generation capability can be used to generate multiple angle of view images corresponding to the target object based on the original image of the target object and the multiple angle of view information, thereby providing conditions for subsequently reconstructing the target object in three dimensions based on the multiple angle of view images to obtain an initial three-dimensional model of the target object. Alternatively, the original image of the target object and the multiple angle of view information can be input into the target image generation model to generate multi-angle images. It should be noted that the multiple angle of view information can be input into the target image generation model M1 at one time, and the target image generation model M1 can generate multiple angle of view images at one time; or one angle of view information can be input into the target image generation model M1 each time, and one angle of view image can be generated each time, and this is not limited. In this embodiment, the multiple angle of view images are generated by the target image generation model M1, and for any angle of view image, the state of the target object that can be observed from the angle of view described by any angle of view information is described. The target image generation model M1 belongs to a pre-trained model, which refers to a model having a multi-angle image generation capability that is pre-trained based on a large amount of sample data. In this embodiment, the pre-trained model having a multi-angle image generation capability can be directly used as the target image generation model M1, or the pre-trained model having a multi-angle image generation capability can be used as an initial image generation model and fine-tuned to obtain the target image generation model M1, and this embodiment is not limited.

[0039] In various application scenarios such as panoramic display, game modeling, 3D commodity display, virtual character display, etc., a three-dimensional model of a target object is needed. Therefore, three-dimensional reconstruction of the target object is needed. In the case where multiple perspective images of the target object are obtained, the target object can be subjected to first-stage three-dimensional reconstruction based on the multiple perspective images to obtain an initial three-dimensional model of the target object.

[0040] Optionally, the manner of three-dimensional reconstruction adopted in the first stage can be explicit three-dimensional reconstruction, such as reconstructing an initial point cloud model or an initial mesh model of the target object. Optionally, in order to better preserve the texture information of the object and improve the model quality of the three-dimensional reconstruction, the manner of three-dimensional reconstruction adopted in the first stage can also be implicit three-dimensional reconstruction, such as reconstructing an initial implicit three-dimensional representation model of the target object, such as a Nerf (Neural Radiance Fields) model. Taking implicit three-dimensional reconstruction of the target object as an example, an initial implicit three-dimensional representation model corresponding to the target object can be created; the process of three-dimensional reconstruction of the target object in the first stage based on the multiple perspective images to obtain the initial implicit three-dimensional representation model of the target object can be referred to the description in subsequent embodiments and will not be described here in detail.

[0041] In this embodiment, in order to improve the model quality of three-dimensional reconstruction of the target object, the entire three-dimensional reconstruction process further includes a second stage; in the second stage, a text-to-image model M2 is configured, and the text-to-image model M2 can be a pre-trained model, i.e., a model having the ability to generate images based on a given prompt word obtained by pre-training based on a large amount of sample data. In order to enable the text-to-image model M2 to generate high-quality images related to the target object, in this embodiment, before the text-to-image model M2 is used, the text-to-image model M2 is first used to learn the association between the target object and the set marker information according to the multiple perspective images of the target object and an initial prompt word containing the set marker information.

[0042] The initial prompt word is used to guide the text-to-image model to generate an image containing the target object as expected by the user. In this embodiment, the set marker information is used to represent the target object in the multiple perspective images and is responsible for identifying the identity attribute of the target object, and in specific implementation, the set marker information can be implemented as a special identifier.

[0043] Wherein, when learning the association relationship between the target object and the set mark information by using the text-to-image model M2, the initial text-to-image model can be fine-tuned according to the multiple perspective images of the target object and the initial prompt words containing the set mark information, that is, the model parameters in the initial text-to-image model are updated, and the target text-to-image model obtained in this way can learn the association relationship between the target object and the set mark information. Wherein, the text-to-image model before fine-tuning is called the initial text-to-image model, and the text-to-image model after fine-tuning is called the target text-to-image model. In other words, the electronic device can fine-tune the text-to-image model M2 by using the multiple perspective images of the target object, so that the text-to-image model M2 associates the target object contained in the multiple perspective images with the set mark information in the initial prompt words, so that the text-to-image model M2 can associate the target object based on the prompt words of the set mark information, and then generate an image containing the target object and meeting the requirements of the prompt words containing the set mark information.

[0044] In this embodiment, the format of the prompt word is not limited, and any prompt word format containing the preset mark information is applicable to the embodiment of the application. In addition, the prompt word can contain other information in addition to the preset mark information, such as object category information and / or scene information, etc., which can be determined according to application requirements.

[0045] In this embodiment, based on learning the association relationship between the target object and the set mark information by using the text-to-image model M2, a scene prompt word containing the set mark information is designed, that is, the scene prompt word also contains scene information of the target object. Based on this, according to the multiple scene prompt words containing the set mark information, the association relationship between the target object and the set mark information learned by using the text-to-image model M2 is used to generate multiple scene images corresponding to the target object. Wherein, the scene prompt word is used to guide the text-to-image model M2 to generate images of the target object in different scenes described by the multiple scene prompt words. In order to facilitate description and distinction, the images of the target object in each scene are called scene images. For example, assuming that the target object is a virtual person, the multiple scene images corresponding to the target object generated by the text-to-image model M2 which has learned the association relationship between the target object and the set mark information based on the multiple scene prompt words containing the set mark information can be: an image of the virtual person standing in front of a sales counter (i.e. a sales scene), an image of the virtual person standing in front of a tourist attraction (i.e. a tourist scene), and so on, an image of the virtual person standing in front of a wardrobe (i.e. a virtual dressing scene), an image of the virtual person speaking, and so on.

[0046] Since the text-to-image model M2 is learned based on multiple perspective images of the target object, the multiple scene images of the target object generated by the text-to-image model M2 learned to associate the target object and the set label information have higher image accuracy, higher resolution, and more image details than the multiple perspective images. Based on this, the initial three-dimensional model can be optimized according to the multiple scene images with higher image accuracy and higher resolution, so as to obtain a target three-dimensional model with higher resolution and clearer model details.

[0047] In the present embodiment, the electronic device can perform multi-stage (two-stage) three-dimensional reconstruction based on a single image of the target object. In the first stage, multiple perspective images are generated based on the image generation model, and the initial three-dimensional model is reconstructed based on the generated multiple perspective images. In the second stage, based on the multiple perspective images and the initial prompt word containing the set label information, the text-to-image model M2 is used to learn the association between the target object and the set label information, and based on this, multiple scene images are generated. Compared with the multiple perspective images, the scene images generated in the second stage have higher resolution and more image details. Furthermore, the initial three-dimensional model is optimized based on the multiple scene images, and a target three-dimensional model with higher resolution and clearer model details can be obtained, which provides conditions for the landing application of the single-image-based three-dimensional reconstruction scheme.

[0048] In the present embodiment, the model structure of the target image generation model M1 is not limited, and any model structure with multiple perspective image generation capability is suitable for the present embodiment. In some optional embodiments, as shown in FIG. 2a, a model structure of the target image generation model M1 is schematically given, which includes an information encoder M11, a diffusion generator M12, and an image decoder M13. Correspondingly, when generating multiple perspective images corresponding to the target object based on the original image of the target object and multiple perspective information using the target image generation model M1, the following steps 111-113 can be implemented:

[0049] Step 111, input the original image and the plurality of perspective information into an information encoder M11 in the target image generation model M1 respectively for vector embedding to obtain the semantic vectors corresponding to the original image and the plurality of perspective information respectively. The information encoder in the target image generation model M1 can be implemented by using a CILP (Contrastive Language-Image Pre-Training) model, or can be implemented by using other models such as a Multimodal Transformers model, and the present embodiment is not limited thereto. The information encoder M11 can include an image encoder M111 and a text encoder M112. The electronic device can input the original image into the image encoder M111 for vector embedding processing, and input the plurality of perspective information into the text encoder M112 for vector embedding processing. It is explained that the plurality of perspective information can be input into the text encoder M112 for vector embedding processing to obtain the semantic vectors corresponding to the plurality of perspective information respectively; or the plurality of perspective information can be input into the text encoder M112 in batches, and each time one perspective information is input into the text encoder M112 for vector embedding processing to obtain the semantic vector corresponding to the perspective information respectively.

[0050] Step 112, input the semantic vectors corresponding to the original image and the plurality of perspective information as generation conditions into a diffusion generator M12 in the image generation model for denoising processing to obtain the feature vectors corresponding to the target object under the plurality of perspective information. The diffusion generator M12 is an image generator based on a diffusion process, and has the following capability: being able to continuously perform denoising processing on a random noise vector under the guidance of the input generation conditions, and finally obtaining a feature vector meeting the requirements of the generation conditions. In the present embodiment, the semantic vector corresponding to the original image and the semantic vector corresponding to each perspective information are input into the diffusion generator M12 as generation conditions of the diffusion generator M12, so that the diffusion generator M12 can continuously perform denoising processing on the random noise vector under the guidance of the original image of the target object and each perspective information, and finally generate the feature vector corresponding to the target object under each perspective information more accurately. In the present embodiment, the number of times of denoising the random noise vector is not limited, for example, it can be 30-50 times, and in each denoising process, the semantic vectors of the original image and the perspective information are continuously injected into the noise vector, so as to obtain a feature vector with rich semantic information, which can also be referred to as a latent space vector.

[0051] In the diffusion process-based image generation, the diffusion stage and the inverse diffusion stage are mainly included. In the diffusion stage, the diffusion generator M12 can gradually add noise to the original image of the target object until the original image is almost completely filled with noise, and a completely noisy image is obtained. Each operation of adding noise is random but controlled within a certain range, ensuring that the entire process is reversible. This process can be regarded as a gradual "diffusion" or "blurring" of the image information of the original image. In the inverse diffusion stage, the diffusion generator M12 can start from the completely noisy image and gradually "denoise" under the guidance of any view information through a series of steps. Specifically, the diffusion generator M12 can predict the noise to be removed in each step and perform the corresponding denoising processing, thereby reversely recovering a clear image that meets the view information, and then the feature vector corresponding to the target object under the view information can be obtained.

[0052] It should be noted that the semantic vectors corresponding to the original image and the plurality of view information can be input into the diffusion generator M12 in the image generation model M1 at the same time, and parallel processing can be performed in the diffusion generator M12 to obtain the feature vectors corresponding to the target object under the plurality of view information. Alternatively, the semantic vectors corresponding to the plurality of view information can be input into the diffusion generator M12 in the image generation model M1 one by one, and the feature vector corresponding to the target object under one view information can be obtained each time until the feature vectors corresponding to the target object under the plurality of view information are obtained.

[0053] In step 113, the feature vectors corresponding to the plurality of view information are input into the image decoder M13 in the image generation model M1 for decoding processing, respectively, to obtain a plurality of view images corresponding to the target object. The image decoder M13 has learned the mapping relationship between the feature vectors and the corresponding view images through a pre-training process, and based on this, the image decoder M13 can decode the feature vectors corresponding to the plurality of view information based on the mapping relationship to obtain the plurality of view images corresponding to the target object.

[0054] It should be noted that the feature vectors corresponding to the plurality of view information can be input into the image decoder M13 in the image generation model M1 at the same time for decoding processing to obtain the plurality of view images. Alternatively, the feature vectors corresponding to the plurality of view information can be input into the image decoder for decoding processing one by one until the plurality of view images are obtained, and the present embodiment does not limit this.

[0055] In this way, the electronic device can use the target image generation model M1 to generate the plurality of view images corresponding to the target object based on the original image of the target object and the plurality of view information, and the plurality of view images can be generated more accurately.

[0056] The model training process of the target image generation model M1 will be described in detail below in combination with the model architecture shown in FIG. 2b, which can be implemented based on the following steps R1-step R2:

[0057] Step R1, the electronic device can perform model training on the initial image generation model M1' according to the sample image and the plurality of sample perspective information to obtain a plurality of sample perspective images.

[0058] Step R2, at least calculate the pixel consistency loss function between the sample image and the plurality of sample perspective images, update the model parameters of the initial image generation model M1' if the pixel consistency loss function does not meet the first preset loss condition, and continue to perform the model training process of step R1 until the pixel consistency loss function meets the first preset loss condition to obtain the target image generation model M1.

[0059] Specifically, step R1 can include the following steps R11-step R16:

[0060] Step R11, input the plurality of sample perspective information into the text encoder M112' in the initial image generation model M1' for vector embedding to obtain a semantic vector of the plurality of sample perspective information. Wherein, the text encoder in the initial image generation model M1' can be implemented by CILP model, or other models such as Multimodal Transformers (multimodal encoding) model, which is not limited by the embodiment.

[0061] Step R12, input the sample image into the image encoder M111' in the initial image generation model for feature extraction to obtain a first sample feature vector.

[0062] Step R13, add the original sample noise signal in the first sample feature vector to obtain a second sample feature vector. Wherein, the original sample noise signal can be Gaussian noise signal or random noise signal, etc., which can be set by the user according to the actual design requirements. The original sample noise signal can be in the form of vector, function or matrix, etc., which is not limited by the embodiment. If the original sample noise signal is in the form of vector, the original sample noise signal and the first sample feature vector can be linearly superimposed to obtain the second sample feature vector; if the original sample noise signal is in the form of function or matrix, the original sample noise signal can be converted to vector form and linearly superimposed with the first sample feature vector to obtain the second sample feature vector.

[0063] Step R14, input the semantic vectors of the plurality of sample view information and the second sample feature vector into the diffusion generator M12' in the initial image generation model to perform noise prediction to obtain a plurality of predicted sample noise signals corresponding to the plurality of sample view information. In other words, this step aims to use the diffusion generator M12' to predict the original sample noise signal added in the second sample feature vector in step R13 to obtain a predicted sample noise signal. Ideally, the predicted sample noise signal should be the original sample noise signal, but usually the two are not the same.

[0064] Step R15, subtract the second sample feature vector from the plurality of predicted sample noise signals to obtain a plurality of third sample feature vectors. Since the second sample feature vector is a vector obtained after adding the original sample noise signal, subtracting the second sample feature vector from the plurality of predicted sample noise signals can restore the state of the second sample feature vector before adding the original sample noise signal, that is, the third sample feature vector. It is explained that usually, the predicted sample noise signal and the original sample noise signal have differences, so the third sample feature vector and the second sample feature vector also have certain differences, of course, in the ideal case, the third sample feature vector and the second sample feature vector are the same. Alternatively, the electronic device can train the diffusion generator M12' under the supervision of a noise loss function, which can be used to describe the error between the third sample feature vector and the second sample feature vector. Specifically, the electronic device can update the parameters of the diffusion generator M12' when the noise loss function converges to a specified range, and stop training when the noise loss function converges to the specified range, thereby continuously improving the noise prediction accuracy of the diffusion generator M12'.

[0065] Step R16, input the plurality of third sample feature vectors into the image decoder M13' in the initial image generation model to obtain a plurality of sample view images. The image decoder M13' can map the plurality of third sample feature vectors back to the original image space, that is, reconstruct the image. Specifically, the image decoder can restore the details and structures of the image layer by layer by performing a series of nonlinear transformations and upsampling operations on the plurality of third sample feature vectors, and finally generate a plurality of sample view images.

[0066] In the model training process described in steps R11 to R16, the initial image generation model M1' is used to obtain the plurality of sample perspective images based on the sample image and the plurality of sample perspective information more accurately. On this basis, the loss function used in the model training can be calculated, and the model parameters of the initial image generation model M1' are updated in the case that the loss function does not satisfy the set loss condition, and the model training of the initial image generation model M1' is continued until the model training process is terminated when the loss function satisfies the set loss condition, thereby obtaining the target image generation model M1. Wherein, the information encoder M11' after training is the information encoder M11 in the target image generation model M1, the image encoder M111' after training is the image encoder M111 in the target image generation model M1, the image encoder M112' after training is the image encoder M112 in the target image generation model M1, the diffusion generator M12' after training is the diffusion generator M12 in the target image generation model M1, and the image encoder M13' after training is the image encoder M13 in the target image generation model M1.

[0067] In this embodiment, the mean square error function between the predicted sample noise signal and the original sample noise signal can be used as the loss function (referred to as the loss function between noises), and the model training of the initial image generation model is continued in the case that the loss function does not satisfy the fourth preset loss condition. For example, when the mean square error function between the predicted sample noise signal and the original sample noise signal is greater than the set mean square error threshold, it is considered that the loss function does not satisfy the fourth preset loss condition, and the model training needs to be continued until the mean square error function between the predicted sample noise signal and the original sample noise signal is less than or equal to the set mean square error threshold. However, since the loss function only focuses on the deviation between the noises, and ignores the deviation between the sample perspective images generated by the model and the sample images, the target image generation model M1 obtained based on the loss function has relatively poor effect in generating the plurality of perspective images, and the plurality of perspective images generated have the defect of pixel inconsistency between the original images, which further affects the effect of three-dimensional reconstruction based on the plurality of perspective images.

[0068] In order to further improve the perspective image generation effect of the image generation model, the pixel consistency loss function is also added in this embodiment, and the initial image generation model is trained in combination with the pixel consistency loss function, so as to obtain the target image generation model M1 with better perspective image generation effect, and further improve the quality of the three-dimensional model obtained by three-dimensional reconstruction based on the plurality of perspective images generated by the target image generation model M1, and solve the problems of pixel inconsistency and possible abnormal pixel points such as spots in the three-dimensional model. The following will be further described in combination with step R2:

[0069] In step R2, at least a pixel consistency loss function between the sample image and the plurality of sample view images is calculated, in a case where the pixel consistency loss function does not satisfy a first preset loss condition, the model parameter of the initial image generation model M1' is updated, and the model training of the initial image generation model M1' is continued to obtain the target image generation model M1. The first preset loss condition can be customized according to actual design requirements, and the embodiment is not limited. The pixel consistency loss function can be calculated based on the following formula:

[0070] L ref =||x⊙m-G θ (B ref )||1 (Formula 1)

[0071] Wherein, x is the sample view image generated by the initial image generation model M1', m is the position coordinates of the foreground segmentation region of the sample view image in the sample view image, the symbol ⊙ represents the dot product, and x⊙m is the dot product result between x and m. G θ is an affine transformation matrix, B ref is the foreground segmentation region in the sample image, and the double vertical bar symbol represents a MSE (Mean Squared Error) loss. Hereinafter, in order to facilitate description, the foreground segmentation region in the sample image is referred to as the first foreground segmentation region, and the foreground segmentation region of the sample view image is referred to as the second foreground segmentation region.

[0072] The calculation process of the pixel consistency loss function expressed by the above formula 1 will be described in the following steps. The calculation process can be implemented based on the following steps R21-step R23:

[0073] Step R21, according to the affine transformation matrix corresponding to the plurality of sample view information, the mapping image of the first foreground segmentation region in the sample image under the plurality of sample view information is generated, and any mapping image is G θ (B ref ) in formula 1. Wherein, each sample view information can correspond to an affine transformation matrix, which is used to map the foreground image to the mapping image under the sample view. Wherein, the electronic device can generate the mapping image of the first foreground segmentation region in the sample image under the plurality of sample view information according to the affine transformation matrix corresponding to the plurality of sample view information; or can generate the mapping image of the first foreground segmentation region in the sample image under each sample view information according to the affine transformation matrix corresponding to each sample view information respectively, and the embodiment is not limited.

[0074] Step R22, extracting the second foreground segmentation region in the plurality of sample view images according to the pixel coordinates of the first foreground segmentation region in the sample image. The second foreground segmentation region in any sample view image obtained by the extraction is x m in formula 1.

[0075] Step R23, calculating the pixel consistency loss function between the mapping image of the first foreground segmentation region under the plurality of sample view information and the second foreground segmentation region in the plurality of sample view images, that is, L in formula 1. ref . Wherein, the pixel consistency loss function can be implemented as the MSE loss mentioned in the foregoing formula 1, and can also be implemented as other losses, such as MAE (Mean Absolute Error, Mean Absolute Error) loss, etc., which are not limited in the embodiment.

[0076] In this way, the electronic device can more accurately calculate the pixel consistency loss function, so as to update the model parameters of the initial image generation model M1' based on the pixel consistency loss function, and continuously improve the view image generation effect of the initial image generation model M1', so that the target image generation model M1 after training can solve the defect that the pixel inconsistency exists between the view image and the original image; and further improve the quality of the three-dimensional model obtained by three-dimensional reconstruction based on the plurality of view images generated based on the target image generation model M1, and solve the pixel inconsistency and possible abnormal pixel point such as spot and other bad generation effect problems existing in the three-dimensional model.

[0077] In some optional embodiments, the step 12 "performing three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial three-dimensional model of the target object" in the foregoing embodiments can be implemented based on the following steps:

[0078] Step 121, performing neural radiance field based three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial implicit three-dimensional representation model of the target object. Wherein, the surface points on the target object correspond to the pixel points in the plurality of view images, and the view information corresponding to the plurality of view images. Wherein, the initial implicit three-dimensional representation model can perform implicit three-dimensional representation on the target object, for example, can represent the shape, texture, material and other dimensional object information of the target object. In the embodiment, the initial implicit three-dimensional representation model can be a fully connected neural network, also known as Multi-Layer Perceptron (MLP). The initial implicit three-dimensional representation model predicts the volume density and color information of the surface points based on the spatial coordinates of the surface points and the view information. Wherein, the initial implicit three-dimensional representation model can be expressed as: σ, c = F (d, x) (formula 2)

[0079] wherein x = (x, y, z), x denotes the spatial coordinates (x, y, z) of the surface point; d denotes the viewing angle information of the surface point is the azimuth angle, is the elevation angle. c = (r, g, b), c denotes the color information (r, g, b) of the spatial point, r refers to red (Red, R), g refers to green (Green, G), and b refers to blue (Blue, B). σ denotes the volume density of the surface point.

[0080] In practical applications, the initial implicit three-dimensional representation model includes an Fσ network for predicting the volume density σ and an Fc network for predicting the color information c. Therefore, the initial implicit three-dimensional representation model can be further expressed as: Fσ: x→(σ, f) (Formula 3) Fc: (d, f)→c (Formula 4)

[0081] It is worth noting that the Fσ network inputs the spatial coordinates x of the surface point and outputs the volume density σ and the intermediate feature f of the surface point. The Fc network inputs the intermediate feature f and the viewing angle information d of the surface point and inputs the color information RGB value of the surface point. That is, the volume density is only related to the spatial coordinates x, and the color information RGB value is related to the spatial coordinates and the viewing angle information.

[0082] In this way, an initial implicit three-dimensional representation model containing more dimensional and richer object information can be generated.

[0083] Step 122, inputting the original image into the initial implicit three-dimensional representation model to generate an image under a specified viewing angle to obtain a specified viewing angle image. Wherein, the implicit three-dimensional representation model can be used to perform volume sampling and density / color interpolation under the specified viewing angle, simulate the process of light passing through the three-dimensional space and interacting with the scene, and thus generate the specified viewing angle image.

[0084] Step 123, calculating a depth loss function and / or a SDS (Score Distillation Sampling) loss function between the specified viewing angle image and the original image, updating the model parameters of the initial implicit three-dimensional representation model until the depth loss function and / or the SDS loss function meets a second preset loss condition, if the depth loss function and / or the SDS loss function does not meet the second preset loss condition. Wherein, the second preset loss condition can be set according to actual design requirements, and the present embodiment is not limited.

[0085] In the embodiments of the present application, in order to avoid the problems of surface depression or excessive flatness of the generated three-dimensional model in the geometric level, a depth loss function between the specified view image and the original image can be introduced, and if the depth loss function does not satisfy the second preset loss condition, the model parameters of the initial implicit three-dimensional representation model are updated and the model training is continued until the depth loss function satisfies the second preset loss condition.

[0086] In addition, although the introduction of the pixel consistency loss function in the target image generation model M1 training process can improve the quality of the multi-view images generated by the target image generation model M1, the pixel-level optimization is unstable, and the generated view images are prone to have illogical results, which affects the quality of the initial implicit three-dimensional representation model generated based on the multi-view images. Therefore, the SDS loss function between the specified view image and the original image can also be introduced, and if the SDS loss function does not satisfy the second preset loss condition, the model parameters of the initial implicit three-dimensional representation model are updated until the SDS loss function satisfies the second preset loss condition to obtain an initial implicit three-dimensional representation model with high quality. The SDS loss function can be used to promote the multi-view consistency of the three-dimensional scene, help the initial implicit three-dimensional representation model learn more accurate three-dimensional spatial distribution, and improve the realism and integrity of the initial three-dimensional model. In the process of generating a three-dimensional scene, the SDS loss function emphasizes the distillation of the scores of the generated images under different views, that is, the initial implicit three-dimensional representation model updates the model parameters under the supervision of the SDS loss function, which not only makes the generated images look more realistic, but also improves the consistency and continuity of the generated images under various randomly sampled views.

[0087] It is explained that the SDS loss function and the depth loss function can be used alternatively or jointly, and the SDS loss function and the depth loss function each correspond to a second preset loss condition, and the second preset loss conditions corresponding to the two are different.

[0088] Specifically, the electronic device can calculate the depth loss function between the specified view image and the original image based on the following steps P1-P2:

[0089] Step P1, respectively, obtain the depth map corresponding to the specified view image and the original image. Among them, the depth map extraction method includes but is not limited to: MiDaS (Multi-View Depth Estimation based on Stereo Matching and Deep Learning, a multi-view depth estimation model based on stereo matching and deep learning technology), LeReS (Learning Residual for Single Image Depth Estimation, a deep learning model applied to the field of monocular depth estimation) and Zoe (a method combining metric depth estimation and relative depth estimation), this embodiment is not limited. Among them, MiDaS is a neural network-based depth estimation method, which aims to infer the distance information of objects in the scene from the image, and can realize the mapping from the input image to the depth map through the encoder and decoder two main components. Among them, LeReS depth information estimation is also a depth estimation method for detecting the depth of field and light shadow of the picture, but its accuracy is higher than that of MiDaS, and the output time is also longer. Zoe depth information calculation is a method combining metric depth estimation and relative depth estimation, which can be used to estimate the depth information of each pixel in the image, and can transfer the depth information of the existing data set to the new target data set, thereby realizing zero-shot (Zero-shot) depth estimation. In order to facilitate description and distinction, the depth map corresponding to the specified view image is denoted as the first depth map, and the depth map corresponding to the original image is denoted as the second depth map.

[0090] Step P2, according to the first depth map corresponding to the specified view image and the second depth map corresponding to the original image, calculate the depth loss function between the specified view image and the original image. Optionally, the absolute error or squared error between the first depth map and the second depth map can be calculated as the depth loss function between the specified view image and the original image. Optionally, the depth loss function can be determined based on the following formula.

[0091] Where d(β ref ) is the depth represented by the first depth map of the specified view image, Cov is the covariance matrix, Var is the standard deviation, d is the true depth represented by the second depth map of the original image, L depth is the depth loss function.

[0092] Through the above steps P1-P2, the electronic device can more accurately determine the depth loss function between the specified view image and the original image.

[0093] Specifically, the electronic device can calculate the SDS loss function between the specified view image and the original image based on the following steps K1-K5.

[0094] Step K1: Input the image from the specified viewpoint into the image encoder in the noise prediction model to extract features and obtain the first initial feature vector.

[0095] Step K2: Add a first original noise signal to the first initial feature vector to obtain a first noisy feature vector. The first original noise signal can be a Gaussian noise signal, random noise signal, etc., and can be set by the user according to actual design requirements. The first original noise signal can be in vector form, function form, matrix form, etc., and this embodiment does not impose any restrictions. If the first original noise signal is in vector form, it can be linearly superimposed with the first initial feature vector to obtain the first noisy feature vector; if the first original noise signal is in function form or matrix form, it can be converted into vector form and linearly superimposed with the first original noise signal to obtain the first noisy feature vector.

[0096] For example, suppose the first initial feature vector is x render With a time step of t, based on this, the first initial feature vector x can be... rende x makes some changes a t And add the first original noise signal σ t ε, yielding the first noisy feature vector: a t x render +σ t ε.

[0097] Step K3: Input the first noisy feature vector into the diffusion generator in the noise prediction model to perform noise prediction, so as to obtain the first predicted noise signal. In other words, this step aims to use the diffusion generator in the noise prediction model to predict the first original noise signal added to the first noisy feature vector in step K2, so as to obtain the first predicted noise signal.

[0098] Step K4: Calculate the mean squared error loss function between the first original noise signal and the first predicted noise signal. The mean squared error loss function includes the gradient of the initial implicit three-dimensional representation model, the gradient of the noise prediction model, and noise contrast information.

[0099] Using the previous example, the mean squared error loss function L can be expressed as: L=E[w(t)||UNet(a t x render +σ t ε|t)-ε|| 2 ]

[0100] Where w(t) is a weighting function with respect to time step t, E represents the mean square error function, and x render As the first initial feature vector, UNet(a t xrender +σ t ε|t) is a first predicted noise signal obtained by the noise prediction model, and ε is a first original noise signal.

[0101] Wherein, the gradient of the above mean square error loss function L can be decomposed into the gradient of the noise contrast information, the gradient of the noise prediction model and the gradient of the initial implicit three-dimensional representation model by the chain rule. The chain rule is a derivative rule in calculus, which is used to find the derivative of a composite function.

[0102] Step K5, obtaining the sum of the gradient of the initial implicit three-dimensional representation model and the noise contrast information as the SDS loss function between the specified view image and the original image. Wherein, considering that the gradient of the noise prediction model contained in the mean square error loss function is difficult to calculate, and its importance is lower in the actual scene application, the sum of the gradient of the initial implicit three-dimensional representation model and the noise contrast information can be directly used as the SDS loss function between the specified view image and the original image.

[0103] Through the above steps K1-K5, the fractional distillation sampling SDS loss function between the specified view image and the original image can be calculated more accurately.

[0104] It is explained that the noise prediction model can be implemented by the target image generation model M1 or the initial image generation model in the above embodiments, but is not limited thereto. For the related description of the image encoder and the diffusion generator in the noise prediction model, please refer to the same description in the foregoing embodiments, which will not be repeated here.

[0105] In the embodiments of the present application, the model architecture of the text-to-image model M2 is not limited, and any model architecture that can generate corresponding images according to given prompt words is applicable to the embodiments of the present application. In some optional embodiments, as shown in FIG. 3, a model architecture of the text-to-image model M2 is illustratively given, which includes a text encoder M21, an image encoder M22, a diffusion generator M23 and an image decoder M24.

[0106] The step 13 in the foregoing embodiments will be further described below in combination with the architecture of the above text-to-image model M2. The step 13 can be implemented based on the following steps 131-136:

[0107] Step 131, generate an initial prompt word in a prompt word format containing class information and label information, the initial prompt word including class information of the target object and set label information. Wherein, the prompt word format containing class information and label information can be a[identifier][class noun], wherein [identifier] is a special marker associated with the target object in the multiple perspective images, that is, the set label information; [class noun] is the class information of the target object, which is used to describe the category to which the target object belongs. Wherein, the class information is used to retain the prior information of the same category as the target object, so as to utilize the prior knowledge about the items in this category in the process of fine-tuning the text-to-image model. For example, if the set label information is used to mark that the target object is a golden retriever, then the class information can be used to describe that the target object belongs to the category of dogs. Therefore, based on the initial prompt word, the process of fine-tuning the text-to-image model can not only utilize the prior knowledge of golden retrievers, but also utilize the prior knowledge of other dogs in the category of "dogs", so that the text-to-image model can be fine-tuned more accurately based on these rich prior knowledge.

[0108] Step 132, input the initial prompt word into the text encoder M21 in the text-to-image model M2 for vector embedding to obtain a semantic vector of the initial prompt word.

[0109] Step 133, input the multiple perspective images into the image encoder M22 in the text-to-image model for feature extraction to obtain a second initial feature vector. It is explained that the multiple perspective images can be input into the image encoder M22 in the text-to-image model M2 for feature extraction at one time, or one perspective image can be input into the image encoder M22 in the text-to-image model M2 for feature extraction at a time, and this is not limited.

[0110] Step 134, add a second original noise signal to the second initial feature vector to obtain a second noise-added feature vector. Wherein, the process of adding the noise signal is the same as the process of adding the noise signal in the foregoing embodiment, which will not be repeated here.

[0111] Step 135, input the semantic vector of the initial prompt word and the second noise-added feature vector into the diffusion generator M23 in the text-to-image model M2 for noise prediction to obtain a second predicted noise signal. In other words, this step aims to predict the second original noise signal added in the second noise-added feature vector in step 134 by using the diffusion generator M23 in the text-to-image model M2 to obtain a second predicted noise signal.

[0112] Step 136, if the loss function between the second original noise signal and the second predicted noise signal does not satisfy the third preset loss condition, update the model parameters of the diffusion generator M23 in the text-to-image model M2 until the third preset loss condition is satisfied. The loss function between the second original noise signal and the second predicted noise signal can be a mean square error loss function, or an average absolute error, etc., which can be set according to actual design requirements, and the present embodiment is not limited. The third preset loss condition can be customized according to actual design requirements, and the present embodiment is not limited. When the text-to-image model that satisfies the loss condition is obtained through fine-tuning, it means that the text-to-image model has successfully learned the association between the target object and the set label information.

[0113] It is explained that in the above model training process, it also includes: subtracting the second noise feature vector from the second predicted noise signal to obtain a de-noised feature vector; inputting the de-noised feature vector into the image decoder M24 for decoding processing to obtain an image meeting the requirements of the initial prompt word. With the continuous fine-tuning process, the decoding processing capability of the image decoder M24 will also be continuously optimized.

[0114] In this way, the text-to-image model can be fine-tuned more accurately according to the multiple perspective images and the initial prompt word containing the set label information, so as to learn the association between the target object and the set label information by using the text-to-image model.

[0115] In some optional embodiments, in combination with the model architecture shown in FIG. 4, when generating multiple scene images corresponding to the target object according to multiple scene prompt words containing set label information by using the association between the target object and the set label information learned by the text-to-image model M2, the following steps can be used:

[0116] Step 141, input the multiple scene prompt words containing the set label information into the text encoder M21 in the text-to-image model M2 for vector embedding to obtain semantic vectors corresponding to the multiple scene prompt words. The text encoder M21 in the text-to-image model M2 has the ability to map the scene prompt words in text form to the semantic space to obtain the corresponding semantic vectors after pre-model training.

[0117] Step 142, input the semantic vectors corresponding to the plurality of scene prompt words as a generation condition into the diffusion generator M23 in the text-to-image model M2 for denoising processing to obtain a plurality of feature vectors corresponding to the plurality of scene prompt words. The diffusion generator M23 in the text-to-image model M2 has the following ability after a pre-training process: according to the guidance of the input generation condition, continuously denoising a random noise vector to obtain a feature vector that meets the generation condition requirement. Based on this model capability, the diffusion generator can be used to more accurately determine the feature vectors corresponding to the plurality of scene prompt words. In this embodiment, the semantic vectors corresponding to the plurality of scene prompt words are input into the diffusion generator M23 as the generation condition of the diffusion generator M23, so that the diffusion generator M23 can continuously denoise the random noise vector guided by the semantic vector corresponding to each scene prompt word, and finally more accurately generate the feature vectors corresponding to the plurality of scene prompt words. In this embodiment, the number of times of denoising the random noise vector is not limited, for example, it can be 30-50 times. In each denoising process, the semantic vector corresponding to the scene prompt word is continuously injected into the noise vector, so as to obtain a feature vector with rich semantic information, which can also be called a latent space vector.

[0118] Step 143, input the plurality of feature vectors into the image decoder of the text-to-image model for decoding processing to obtain a plurality of scene images corresponding to the target object. The image decoder of the text-to-image model has learned the mapping relationship from the feature vector to the corresponding scene image through a pre-training process, and based on this, the image decoder of the text-to-image model can decode the feature vectors corresponding to the plurality of scene prompt words based on the mapping relationship to obtain a plurality of scene images corresponding to the target object.

[0119] In this way, the plurality of scene images generated by the text-to-image model that has learned the association between the target object and the setting label information have higher image accuracy and higher resolution than the plurality of perspective images.

[0120] Based on this, in some optional embodiments, the initial three-dimensional model can be optimized according to the plurality of scene images with higher image accuracy and higher resolution, so as to obtain a target three-dimensional model with higher resolution and clearer model details. In an optional embodiment, in the case where the initial three-dimensional model is implemented as an initial implicit three-dimensional representation model, a neural network-based three-dimensional reconstruction can be performed according to the plurality of scene images and the initial implicit three-dimensional representation model to obtain a target implicit three-dimensional representation model of the target object.

[0121] Wherein, since the resolution of the scene image is higher and the image details are more abundant relative to the multiple view images, the target implicit three-dimensional representation model obtained based on the neural network three-dimensional reconstruction based on the multiple scene images and the initial implicit three-dimensional representation model has stronger ability to express the target object in an implicit three-dimensional manner, and can express more abundant and accurate object information of multiple dimensions such as shape, texture, and material of the target object.

[0122] On the basis of each of the foregoing embodiments, before optimizing the initial three-dimensional model according to the multiple scene images corresponding to the target object to obtain the target three-dimensional model of the target object, as shown in FIG. 1b, the image inpainting model M3 can also be used to perform block-by-block inpainting on any scene image, thereby improving the resolution of the scene image and enriching the image details of the scene image.

[0123] Optionally, the structure of the image inpainting model M3 at least includes a segmentation module M31, a semantic recognition module M32, a detail inpainting module M33, and a splicing module M34. Based on the structure of the image inpainting model M3, the process of using the image inpainting model M3 to perform block-by-block inpainting on any scene image can be implemented as follows: for any scene image, the scene image is input into the segmentation module M31 in the image inpainting model M3 for image segmentation, so as to segment the scene image into multiple image blocks, and the multiple image blocks are input into the semantic recognition module M32 in the image inpainting model M3 for semantic recognition, so as to obtain semantic information of the multiple image blocks. Since the semantic recognition module M32 can process each image block individually, the semantic information of each image block can be extracted more accurately. Then, the semantic information of the multiple image blocks can be input into the detail inpainting module M33 in the image inpainting model M3 for semantic-based image detail inpainting, so as to obtain multiple image blocks after inpainting. The detail inpainting module M33 can introduce an attention mechanism in the inpainting process, so as to focus on the context information around the damaged area and extract useful features to guide the inpainting process. The detail inpainting module M33 can be based on a generative adversarial network (GANs), a variational autoencoder (VAEs), or a progressive generative network (PGNs), and generate reasonable detail filling content according to the context information of the damaged area, and use the detail filling content to inpaint the damaged area. After that, the multiple image blocks after modification can be input into the splicing module M34 for splicing, so as to obtain the scene image after inpainting.

[0124] In this way, the block-by-block processing capability of the image inpainting model M3 can be used to process different image blocks individually, such as feature extraction and detail inpainting, so that the resolution of the scene image after final inpainting is higher and the image details are more abundant compared with the scene image before inpainting.

[0125] It is explained that the various models provided by the embodiments of the present application can be implemented using existing models or can be built according to the architecture of each model given by the embodiments of the present application, and no limitation is made thereto. For example, in the case of using existing models, the target image generation model M1 and the noise prediction model in the present embodiment can use, but are not limited to, the Stable Diffusion model, the text-to-image model can use, but is not limited to, the Dreambooth model, and the image inpainting model can use, but is not limited to, the Tile model. Among them, the Stable Diffusion model is based on the mathematical principle of diffusion process, simulates the diffusion and aggregation process of particles in image space, is a variant of generative adversarial network (GAN), but uses a more stable and controllable diffusion mechanism to generate images, thereby improving the quality and stability of the generated images while maintaining creativity. The Dreambooth model is mainly used to fine-tune pre-trained large visual models (such as Transformer-based models, Dreambooth models, etc.) so that these pre-trained models can generate or recognize image content for specific personalized needs. The core of this technology is to let the pre-trained model "learn" to focus on specific objects or scenes defined by the user, so as to reflect this personalized understanding in subsequent image generation or classification tasks. The Tile model is used to improve processing efficiency or reduce memory occupancy when processing large-scale image data, which can use the method of dividing large images into multiple small blocks (Tiles) for processing respectively, and then merging the results to complete relatively complex image processing tasks.

[0126] The three-dimensional model reconstruction method in the above embodiments can also be applied to various scenes, such as game scenes, panoramic display scenes, game modeling scenes, 3D commodity display scenes, virtual character display scenes, and network shopping scenes. The following will be described in detail by scene:

[0127] In the game scene, in response to a generation operation of a user on a virtual game item (i.e., a target object), an original image of the virtual game item can be obtained from a local material library. For example, the user wants to generate a chair in a virtual game world, and the user can click a generation control for the virtual game item. Correspondingly, based on the original image of the virtual game item and multiple perspective information, a target image generation model M1 can be used to generate multiple perspective images corresponding to the virtual game item; based on the multiple perspective images, three-dimensional reconstruction is performed on the virtual game item to obtain an initial three-dimensional model of the virtual game item; based on the multiple perspective images and an initial prompt word containing setting mark information, a text-to-image model M2 is used to learn an association relationship between the virtual game item and the setting mark information; based on multiple scene prompt words containing the setting mark information and the association relationship between the virtual game item and the setting mark information learned by the text-to-image model M2, multiple scene images corresponding to the virtual game item are generated; and based on the multiple scene images corresponding to the virtual game item, the initial three-dimensional model is optimized to obtain a target three-dimensional model of the virtual game item.

[0128] In the network shopping scene, after a user completes a shopping operation on an actual commodity through a terminal device held by the user, the terminal device can send a virtual commodity generation instruction corresponding to the actual commodity to a cloud server (i.e., a server in the present application). Then, an original image of the virtual commodity can be obtained from a local material library, and based on the original image of the virtual commodity and multiple perspective information, a target image generation model M1 can be used to generate multiple perspective images corresponding to the virtual commodity; based on the multiple perspective images, three-dimensional reconstruction is performed on the virtual commodity to obtain an initial three-dimensional model of the virtual commodity; based on the multiple perspective images and an initial prompt word containing setting mark information, a text-to-image model M2 is used to learn an association relationship between the virtual commodity and the setting mark information; based on multiple scene prompt words containing the setting mark information and the association relationship between the virtual game item and the setting mark information learned by the text-to-image model M2, multiple scene images corresponding to the virtual game item are generated; and based on the multiple scene images corresponding to the virtual game item, the initial three-dimensional model is optimized to obtain a target three-dimensional model of the virtual game item. In this scene embodiment, the generation of the target three-dimensional model of the virtual commodity by the server is taken as an example for description, but the present application is not limited thereto. With the continuous enhancement of the function of the terminal device, the generation process of the target three-dimensional model can also be implemented on the terminal device.

[0129] Then, the target three-dimensional model can be sent to the terminal device of the user to display the target three-dimensional model in a virtual shopping space displayed by the terminal device. For example, in the virtual shopping space displayed by the terminal device, there are a virtual character and other furniture and furnishings, and when the user purchases a table, the user can open the virtual shopping space of the user, and the target three-dimensional model corresponding to the table can be displayed in the virtual shopping space.

[0130] In a 3D commodity display scenario, a user often has a demand for identifying a corresponding commodity through an image. Specifically, the user can use a shopping type application program running on a terminal device to take a picture of a target object and send a target image containing the target object to a server. The server can determine a corresponding commodity link based on the target image and return it to the terminal device for display on the application program. However, considering that the display of a target object in a three-dimensional form can enable a user to more intuitively understand the target object, the present application proposes a three-dimensional model reconstruction method applicable to a 3D commodity display scenario. Specifically, the terminal device can take a picture of a target object and send a target image containing the target object to a server. The server can generate a plurality of perspective images corresponding to the target object based on the target image of the target object and a plurality of perspective information using a target image generation model M1; perform three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object; learn the association between the target object and the set marker information according to the plurality of perspective images and the initial prompt words containing the set marker information using a text-to-image model M2; generate a plurality of scene images corresponding to the target object according to the association between the target object and the set marker information learned by the text-to-image model M2 according to a plurality of scene prompt words containing the set marker information; and optimize the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object.

[0131] After that, the target three-dimensional model can be sent to the terminal device for the terminal device to display the target three-dimensional model. For example, the terminal device displays a virtual exhibition stand, and a virtual light is provided beside the exhibition stand for a user to adjust the light. After the user takes a picture of a pair of glasses using the camera of the terminal device, the terminal device uploads the image corresponding to the glasses to the server, the server generates a target three-dimensional model of the glasses according to the above method and returns it to the terminal device, and the user can open a three-dimensional display space, which can display the target three-dimensional model corresponding to the glasses. In this embodiment, the generation of the target three-dimensional model of the glasses by the server is taken as an example for illustration, but the present application is not limited thereto. With the continuous enhancement of the functions of the terminal device, the generation process of the target three-dimensional model can also be implemented on the terminal device.

[0132] In addition to the three-dimensional model reconstruction method in the foregoing embodiments, the present application also provides an image generation method. The method can include the steps shown in FIG. 5:

[0133] Step 51: generating a plurality of perspective images corresponding to a target object based on a target image of the target object and a plurality of perspective information using a target image generation model.

[0134] Step 52: performing three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object.

[0135] Step 53, learning the association between the target object and the set mark information by using the text-to-image model according to the multiple view images and the initial prompt word containing the set mark information.

[0136] Step 54, generating multiple scene images corresponding to the target object according to the multiple scene prompt words containing the set mark information and the association between the target object and the set mark information learned by using the text-to-image model.

[0137] In the method, multi-stage three-dimensional reconstruction is performed based on a single image of a target object. In the first stage, multiple view images are generated based on an image generation model, and an initial three-dimensional model is reconstructed based on the multiple view images. In the second stage, an association between the target object and set mark information is learned by using a text-to-image model based on the multiple view images and an initial prompt word containing the set mark information, and multiple scene images are generated based on the learned association. Compared with the multiple view images, the scene images generated in the second stage have higher resolution and more detailed image details.

[0138] Optionally, the multiple view images corresponding to the target object are generated by using a target image generation model based on the original image of the target object and the multiple view information, including: inputting the original image and the multiple view information into an information encoder in the target image generation model for vector embedding to obtain semantic vectors corresponding to the original image and the multiple view information respectively; inputting the semantic vectors corresponding to the original image and the multiple view information respectively as generation conditions into a diffusion generator in the target image generation model for denoising processing to obtain feature vectors corresponding to the multiple view information; and inputting the feature vectors corresponding to the multiple view information into an image decoder in the target image generation model for decoding processing to obtain the multiple view images corresponding to the target object.

[0139] Optionally, the method further includes: training the initial image generation model according to the sample image and the multiple sample view information to obtain multiple sample view images; and calculating at least a pixel consistency loss function between the sample image and the multiple sample view images, and updating model parameters of the initial image generation model to obtain the target image generation model in a case where the pixel consistency loss function does not satisfy a first preset loss condition.

[0140] Optionally, the initial image generation model is trained according to the sample image and the plurality of sample perspective information to obtain a plurality of sample perspective images, including: inputting the plurality of sample perspective information into a text encoder in the initial image generation model for vector embedding to obtain a semantic vector of the plurality of sample perspective information; inputting the sample image into an image encoder in the initial image generation model for feature extraction to obtain a first sample feature vector; adding an original sample noise signal in the first sample feature vector to obtain a second sample feature vector; inputting the semantic vector of the plurality of sample perspective information and the second sample feature vector into a diffusion generator in the initial image generation model for noise prediction to obtain a plurality of predicted sample noise signals corresponding to the plurality of sample perspective information; subtracting the second sample feature vector from the plurality of predicted sample noise signals to obtain a plurality of third sample feature vectors; and inputting the plurality of third sample feature vectors into an image decoder in the initial image generation model to obtain the plurality of sample perspective images.

[0141] Optionally, a pixel consistency loss function between the sample image and the plurality of sample perspective images is calculated, including: generating a mapping image of a first foreground segmentation region in the sample image under the plurality of sample perspective information according to an affine transformation matrix corresponding to the plurality of sample perspective information; extracting a second foreground segmentation region in the plurality of sample perspective images according to a pixel coordinate of the first foreground segmentation region in the sample image; and calculating a pixel consistency loss function between the mapping image of the first foreground segmentation region under the plurality of sample perspective information and the second foreground segmentation region in the plurality of sample perspective images.

[0142] Optionally, a three-dimensional reconstruction is performed on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object, including: performing a neural radiance field-based three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial implicit three-dimensional representation model of the target object; inputting the original image into the initial implicit three-dimensional representation model for image generation under a specified perspective to obtain a specified perspective image; calculating a depth loss function and / or a score distillation sampling (SDS) loss function between the specified perspective image and the original image; and if the depth loss function and / or the SDS loss function do not satisfy a second preset loss condition, updating a model parameter of the initial implicit three-dimensional representation model until the second preset loss condition is satisfied.

[0143] Optionally, the score distillation sampling SDS loss function between the specified view image and the original image is calculated, including: inputting the specified view image into an image encoder in the noise prediction model for feature extraction to obtain a first initial feature vector; adding a first original noise signal in the first initial feature vector to obtain a first noise-added feature vector; inputting the first noise-added feature vector into a diffusion generator in the noise prediction model for noise prediction to obtain a first predicted noise signal; calculating a mean square error loss function between the first original noise signal and the first predicted noise signal, the mean square error loss function including a gradient of the initial implicit three-dimensional representation model, a gradient of the noise prediction model, and noise contrast information; and obtaining a sum of the gradient of the initial implicit three-dimensional representation model and the noise contrast information as the SDS loss function between the specified view image and the original image.

[0144] Optionally, according to the plurality of view images and the initial prompt word containing the set mark information, the association relationship between the target object and the set mark information is learned by using the text-to-image model, including: generating the initial prompt word according to a prompt word format containing category information and mark information, the initial prompt word including the category information of the target object and the set mark information; inputting the initial prompt word into a text encoder in the text-to-image model for vector embedding to obtain a semantic vector of the initial prompt word; inputting the plurality of view images into an image encoder in the text-to-image model for feature extraction to obtain a second initial feature vector; adding a second original noise signal in the second initial feature vector to obtain a second noise-added feature vector; inputting the semantic vector of the initial prompt word and the second noise-added feature vector into a diffusion generator in the text-to-image model for noise prediction to obtain a second predicted noise signal; and if a loss function between the second original noise signal and the second predicted noise signal does not satisfy a third preset loss condition, updating a model parameter of the diffusion generator in the text-to-image model until the third preset loss condition is satisfied.

[0145] Optionally, according to the plurality of scene prompt words containing the set mark information, the association relationship between the target object and the set mark information learned by using the text-to-image model is used to generate a plurality of scene images corresponding to the target object, including: inputting the plurality of scene prompt words containing the set mark information into a text encoder in the text-to-image model for vector embedding to obtain a plurality of semantic vectors corresponding to the plurality of scene prompt words; inputting the plurality of semantic vectors corresponding to the plurality of scene prompt words as a generation condition into a diffusion generator in the text-to-image model for denoising processing to obtain a plurality of feature vectors corresponding to the plurality of scene prompt words; and inputting the plurality of feature vectors into an image decoder of the text-to-image model for decoding processing to obtain the plurality of scene images corresponding to the target object.

[0146] The detailed implementation and beneficial effects of each step in the method of the embodiment have been described in detail in the foregoing embodiments, and will not be described in detail here.

[0147] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 11 to 15 can be device A; for another example, the execution subject of steps 11-13 can be device A, and the execution subject of steps 14-15 can be device B; and the like.

[0148] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a certain order are included, but it should be clearly understood that these operations can be executed in the order appearing in the text or in parallel, and the serial numbers of the operations such as 11, 12, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second", and the like described herein are used to distinguish different messages, devices, modules, etc., and do not represent the order, nor do "first" and "second" represent different types.

[0149] FIG. 6 is a structural schematic diagram of a three-dimensional model reconstruction device provided by another exemplary embodiment of the present application. As shown in FIG. 6, the device includes:

[0150] The first generation module 601 is configured to generate a plurality of perspective images corresponding to the target object based on the original image of the target object and a plurality of perspective information by using a target image generation model; the reconstruction module 602 is configured to perform three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object; the learning module 603 is configured to learn the association between the target object and the set marker information by using a text-to-image model according to the plurality of perspective images and an initial prompt word containing the set marker information; the second generation module 604 is configured to generate a plurality of scene images corresponding to the target object by using the association between the target object and the set marker information learned by the text-to-image model according to a plurality of scene prompt words containing the set marker information; and the optimization module 605 is configured to optimize the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object.

[0151] Optionally, the first generation module 601 is configured to, when generating the multiple view images corresponding to the target object based on the original image of the target object and the multiple view information, specifically configured to: input the original image and the multiple view information into an information encoder in the target image generation model for vector embedding to obtain semantic vectors corresponding to the original image and the multiple view information respectively; input the semantic vectors corresponding to the original image and the multiple view information respectively as generation conditions into a diffusion generator in the target image generation model for denoising processing to obtain feature vectors corresponding to the multiple view information; and input the feature vectors corresponding to the multiple view information into an image decoder in the target image generation model for decoding processing to obtain the multiple view images corresponding to the target object.

[0152] Optionally, the first generation module 601 is further configured to: perform model training on an initial image generation model based on a sample image and multiple sample view information to obtain multiple sample view images; calculate at least a pixel consistency loss function between the sample image and the multiple sample view images, and update model parameters of the initial image generation model to obtain the target image generation model in a case where the pixel consistency loss function does not satisfy a first preset loss condition.

[0153] Optionally, the first generation module 601 is configured to, when performing model training on an initial image generation model based on a sample image and multiple sample view information to obtain multiple sample view images, specifically configured to: input the multiple sample view information into a text encoder in the initial image generation model for vector embedding to obtain semantic vectors of the multiple sample view information; input the sample image into an image encoder in the initial image generation model for feature extraction to obtain a first sample feature vector; add an original sample noise signal to the first sample feature vector to obtain a second sample feature vector; input the semantic vectors of the multiple sample view information and the second sample feature vector into a diffusion generator in the initial image generation model for noise prediction to obtain multiple predicted sample noise signals corresponding to the multiple sample view information; subtract the second sample feature vector from the multiple predicted sample noise signals to obtain multiple third sample feature vectors; and input the multiple third sample feature vectors into an image decoder in the initial image generation model to obtain the multiple sample view images.

[0154] Optionally, when the first generation module 601 calculates the pixel consistency loss function between the sample image and the plurality of sample view images, specifically used for: generating a mapping image of a first foreground segmentation region in the sample image under the plurality of sample view information according to the affine transformation matrix corresponding to the plurality of sample view information; extracting a second foreground segmentation region in the plurality of sample view images according to the pixel coordinates of the first foreground segmentation region in the sample image; calculating the pixel consistency loss function between the mapping image of the first foreground segmentation region under the plurality of sample view information and the second foreground segmentation region in the plurality of sample view images.

[0155] Optionally, when the reconstruction module 602 performs three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial three-dimensional model of the target object, specifically used for: performing neural radiance field-based three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial implicit three-dimensional representation model of the target object; inputting the original image into the initial implicit three-dimensional representation model to generate an image under a specified view to obtain a specified view image; calculating a depth loss function and / or a score distillation sampling (SDS) loss function between the specified view image and the original image; if the depth loss function and / or the score distillation sampling loss function does not satisfy a second preset loss condition, updating the model parameters of the initial implicit three-dimensional representation model until the second preset loss condition is satisfied.

[0156] Optionally, when the reconstruction module 602 calculates the score distillation sampling (SDS) loss function between the specified view image and the original image, specifically used for: inputting the specified view image into an image encoder in a noise prediction model for feature extraction to obtain a first initial feature vector; adding a first original noise signal in the first initial feature vector to obtain a first noise-added feature vector; inputting the first noise-added feature vector into a diffusion generator in the noise prediction model for noise prediction to obtain a first predicted noise signal; calculating a mean square error loss function between the first original noise signal and the first predicted noise signal, the mean square error loss function including a gradient of the initial implicit three-dimensional representation model, a gradient of the noise prediction model, and noise contrast information; obtaining the sum of the gradient of the initial implicit three-dimensional representation model and the noise contrast information as the SDS loss function between the specified view image and the original image.

[0157] Optionally, in a process in which the learning module 603 learns the association between the target object and the set label information according to the multiple view images and the initial prompt word containing the set label information, the learning module 603 is specifically configured to: generate the initial prompt word in a prompt word format containing category information and label information, the initial prompt word including the category information of the target object and the set label information; input the initial prompt word into a text encoder in the text-to-image model for vector embedding to obtain a semantic vector of the initial prompt word; input the multiple view images into an image encoder in the text-to-image model for feature extraction to obtain a second initial feature vector; add a second original noise signal to the second initial feature vector to obtain a second noise-added feature vector; input the semantic vector of the initial prompt word and the second noise-added feature vector into a diffusion generator in the text-to-image model for noise prediction to obtain a second predicted noise signal; and update a model parameter of the diffusion generator in the text-to-image model until a third preset loss condition is met, if a loss function between the second original noise signal and the second predicted noise signal does not satisfy the third preset loss condition.

[0158] Optionally, in a process in which the second generation module 604 generates the multiple scene images corresponding to the target object according to the multiple scene prompt words containing the set label information and the association between the target object and the set label information learned by the text-to-image model, the second generation module 604 is specifically configured to: input the multiple scene prompt words containing the set label information into a text encoder in the text-to-image model for vector embedding to obtain a semantic vector corresponding to the multiple scene prompt words; input the semantic vector corresponding to the multiple scene prompt words as a generation condition into a diffusion generator in the text-to-image model for denoising processing to obtain a feature vector corresponding to the multiple scene prompt words; and input the multiple feature vectors into an image decoder of the text-to-image model for decoding processing to obtain the multiple scene images corresponding to the target object.

[0159] Further optionally, in a process in which the optimization module 605 optimizes the initial three-dimensional model according to the multiple scene images corresponding to the target object to obtain a target three-dimensional model of the target object, the optimization module 605 is specifically configured to: perform neural network-based three-dimensional reconstruction according to the multiple scene images and the initial implicit three-dimensional representation model to obtain a target implicit three-dimensional representation model of the target object.

[0160] Further optionally, before optimizing the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object, the optimization module 605 is further configured to: for any scene image, input the any scene image into a segmentation module in an image inpainting model for image segmentation to segment the any scene image into a plurality of image blocks; input the semantic information of the plurality of image blocks into a detail inpainting module in the image inpainting model for semantic-based image detail inpainting to obtain a plurality of inpainted image blocks; and splice the plurality of inpainted image blocks to obtain an inpainted scene image.

[0161] In this embodiment, multi-stage three-dimensional reconstruction is performed based on a single image of a target object. In a first stage, a plurality of perspective images are generated based on an image generation model, and an initial three-dimensional model is reconstructed based on the generated plurality of perspective images. In a second stage, based on the plurality of perspective images and an initial prompt word containing set marker information, a text-to-image model is used to learn the association between the target object and the set marker information, and based on this, a plurality of scene images are generated. Compared with the plurality of perspective images, the scene images generated in the second stage have higher resolution and more detailed image details. Then, the initial three-dimensional model is optimized based on the plurality of scene images, and a target three-dimensional model with higher resolution and clearer model details can be obtained, which provides conditions for the landing application of the single-image-based three-dimensional reconstruction scheme.

[0162] The internal functions and structures of the three-dimensional model reconstruction apparatus are described above. As shown in FIG. 7, in practice, the three-dimensional model reconstruction apparatus can be implemented as an electronic device, which includes a memory 701, a processor 702, and a communication component 703.

[0163] The memory 701 is configured to store computer programs and can be configured to store other various data to support operations on the computing platform. Examples of these data include instructions of any application program or method for operating on the computing platform, contact data, phonebook data, messages, pictures, videos, etc.

[0164] The memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0165] The processor 702 is coupled with the memory 701 and is configured to execute a computer program in the memory 701 to: generate a plurality of perspective images corresponding to a target object based on an original image of the target object and a plurality of perspective information by using a target image generation model; perform three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object; learn an association relationship between the target object and a set marker information according to the plurality of perspective images and an initial prompt word containing the set marker information by using a text-to-image model; generate a plurality of scene images corresponding to the target object according to the association relationship between the target object and the set marker information learned by the text-to-image model based on a plurality of scene prompt words containing the set marker information; and optimize the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object.

[0166] Optionally, when the processor 702 generates a plurality of perspective images corresponding to a target object based on an original image of the target object and a plurality of perspective information by using a target image generation model, the processor 702 is specifically configured to: input the original image and the plurality of perspective information into an information encoder in the target image generation model for vector embedding to obtain semantic vectors corresponding to the original image and the plurality of perspective information respectively; input the semantic vectors corresponding to the original image and the plurality of perspective information respectively as generation conditions into a diffusion generator in the target image generation model for denoising processing to obtain feature vectors corresponding to the plurality of perspective information; and input the feature vectors corresponding to the plurality of perspective information into an image decoder in the target image generation model for decoding processing respectively to obtain the plurality of perspective images corresponding to the target object.

[0167] Optionally, the processor 702 is further configured to: perform model training on an initial image generation model according to a sample image and a plurality of sample perspective information to obtain a plurality of sample perspective images; calculate at least a pixel consistency loss function between the sample image and the plurality of sample perspective images, and update model parameters of the initial image generation model to obtain the target image generation model in a case where the pixel consistency loss function does not satisfy a first preset loss condition.

[0168] Optionally, when the processor 702 performs model training on the initial image generation model according to the sample image and the plurality of sample view information to obtain a plurality of sample view images, the processor 702 is specifically configured to: input the plurality of sample view information into a text encoder in the initial image generation model for vector embedding to obtain semantic vectors of the plurality of sample view information; input the sample image into an image encoder in the initial image generation model for feature extraction to obtain a first sample feature vector; add an original sample noise signal to the first sample feature vector to obtain a second sample feature vector; input the semantic vectors of the plurality of sample view information and the second sample feature vector into a diffusion generator in the initial image generation model for noise prediction to obtain a plurality of predicted sample noise signals corresponding to the plurality of sample view information; subtract the second sample feature vector from the plurality of predicted sample noise signals to obtain a plurality of third sample feature vectors; and input the plurality of third sample feature vectors into an image decoder in the initial image generation model to obtain the plurality of sample view images.

[0169] Optionally, when the processor 702 calculates a pixel consistency loss function between the sample image and the plurality of sample view images, the processor 702 is specifically configured to: generate, according to an affine transformation matrix corresponding to the plurality of sample view information, a mapping image of a first foreground segmentation region in the sample image under the plurality of sample view information; extract a second foreground segmentation region in the plurality of sample view images according to a pixel coordinate of the first foreground segmentation region in the sample image; and calculate a pixel consistency loss function between the mapping image of the first foreground segmentation region under the plurality of sample view information and the second foreground segmentation region in the plurality of sample view images.

[0170] Optionally, when the processor 702 performs three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial three-dimensional model of the target object, the processor 702 is specifically configured to: perform neural radiance field-based three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial implicit three-dimensional representation model of the target object; input the original image into the initial implicit three-dimensional representation model for image generation under a specified view angle to obtain a specified view image; calculate a depth loss function and / or a score distillation sampling (SDS) loss function between the specified view image and the original image; and if the depth loss function and / or the SDS loss function does not satisfy a second preset loss condition, update a model parameter of the initial implicit three-dimensional representation model until the second preset loss condition is satisfied.

[0171] Optionally, when the processor 702 calculates the score distillation sampling (SDS) loss function between the specified view image and the original image, specifically for: inputting the specified view image into an image encoder in the noise prediction model for feature extraction to obtain a first initial feature vector; adding a first original noise signal in the first initial feature vector to obtain a first noise-added feature vector; inputting the first noise-added feature vector into a diffusion generator in the noise prediction model for noise prediction to obtain a first predicted noise signal; calculating a mean square error loss function between the first original noise signal and the first predicted noise signal, the mean square error loss function comprising gradients of the initial implicit three-dimensional representation model, gradients of the noise prediction model, and noise contrast information; obtaining a sum of the gradients of the initial implicit three-dimensional representation model and the noise contrast information as the SDS loss function between the specified view image and the original image.

[0172] Optionally, when the processor 702 learns the association between the target object and the set label information according to the plurality of view images and the initial prompt word containing the set label information by using the text-to-image model, specifically for: generating the initial prompt word in a prompt word format containing category information and label information, the initial prompt word comprising category information of the target object and the set label information; inputting the initial prompt word into a text encoder in the text-to-image model for vector embedding to obtain a semantic vector of the initial prompt word; inputting the plurality of view images into an image encoder in the text-to-image model for feature extraction to obtain a second initial feature vector; adding a second original noise signal in the second initial feature vector to obtain a second noise-added feature vector; inputting the semantic vector of the initial prompt word and the second noise-added feature vector into a diffusion generator in the text-to-image model for noise prediction to obtain a second predicted noise signal; if a loss function between the second original noise signal and the second predicted noise signal does not satisfy a third preset loss condition, updating model parameters of the diffusion generator in the text-to-image model until the third preset loss condition is satisfied.

[0173] Optionally, when the processor 702 generates the plurality of scene images corresponding to the target object according to the plurality of scene prompt words containing the set mark information and the association relationship between the target object and the set mark information learned by the text-to-image model, the processor 702 is specifically configured to: input the plurality of scene prompt words containing the set mark information into a text encoder in the text-to-image model for vector embedding to obtain semantic vectors corresponding to the plurality of scene prompt words; input the semantic vectors corresponding to the plurality of scene prompt words as a generation condition into a diffusion generator in the text-to-image model for denoising processing to obtain feature vectors corresponding to the plurality of scene prompt words; and input the plurality of feature vectors into an image decoder of the text-to-image model for decoding processing to obtain the plurality of scene images corresponding to the target object.

[0174] Further optionally, when the processor 702 optimizes the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object, the processor 702 is specifically configured to: perform neural network-based three-dimensional reconstruction according to the plurality of scene images and the initial implicit three-dimensional representation model to obtain a target implicit three-dimensional representation model of the target object.

[0175] Further optionally, before the processor 702 optimizes the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object, the processor 702 is further configured to: for any scene image, input the any scene image into a segmentation module in an image inpainting model for image segmentation to segment the any scene image into a plurality of image blocks; input the plurality of image blocks into a semantic recognition module in the image inpainting model for semantic recognition to obtain semantic information of the plurality of image blocks; input the semantic information of the plurality of image blocks into a detail inpainting module in the image inpainting model for semantic-based image detail inpainting to obtain a plurality of modified image blocks; and splice the plurality of modified image blocks to obtain a modified scene image.

[0176] Further, as shown in FIG. 7, the electronic device further includes a display 704, a power supply component 705, and other components. Only some components are shown in FIG. 7, and this does not mean that the electronic device only includes the components shown in FIG. 7. In addition, the components in the dashed box in FIG. 7 are optional components, not mandatory components, and the specific product form of the working node can be determined. The working node of the embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, or an IOT device, or as a server device such as a conventional server, a cloud server, or a server array. If the working node of the embodiment is implemented as a terminal device such as a desktop computer, a notebook computer, or a smart phone, it can include the components in the dashed box in FIG. 7; if the working node of the embodiment is implemented as a server device such as a conventional server, a cloud server, or a server array, it can not include the components in the dashed box in FIG. 7.

[0177] In the embodiment, multi-stage three-dimensional reconstruction is performed based on a single image of a target object. In the first stage, generation of multiple perspective images is performed based on an image generation model, and initial three-dimensional model reconstruction is performed based on the generated multiple perspective images. In the second stage, based on the multiple perspective images and an initial prompt word containing set marker information, an association between the target object and the set marker information is learned using a text-to-image model, and on this basis, multiple scene images are generated. Compared with the multiple perspective images, the scene images generated in the second stage have higher resolution and more detailed image details, and the initial three-dimensional model is optimized based on the multiple scene images, so that a target three-dimensional model with higher resolution and clearer model details can be obtained, thereby providing conditions for the landing application of the single-image-based three-dimensional reconstruction scheme.

[0178] Correspondingly, the embodiment of the application also provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can implement the steps of the three-dimensional model reconstruction method or the image generation method executable by the electronic device.

[0179] The embodiment of the application also provides a computer program product, including computer programs / instructions, when the computer programs / instructions are executed by a processor, the processor can implement the steps of the three-dimensional model reconstruction method or the image generation method executable by the electronic device.

[0180] The above-described memory can be implemented by any type of volatile or nonvolatile memory devices or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0181] The above-described communication component is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G, or the like, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast managing system via a broadcast channel.

[0182] The above-described display includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect a duration and a pressure related to a touch or a slide operation.

[0183] The above-described power component provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0184] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system, or a computer program product. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, Compact Disc Read-Only Memory (CD-ROM), optical memory, and the like) embodying computer readable program code.

[0185] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in the flowchart one or more blocks and / or in the block or blocks of the block diagram.

[0186] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in the flowchart one or more blocks and / or in the block or blocks of the block diagram.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in the flowchart one or more blocks and / or in the block or blocks of the block diagram.

[0188] In one typical arrangement, the computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0189] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), or electrically erasable programmable read only memory (EEPROM), for the storage of software that is read during runtime. The memory is an example of computer readable media.

[0190] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0191] It should also be noted that the terms "comprising", "including", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0192] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method of three-dimensional model reconstruction, characterized by, The method comprises the steps of: generating a plurality of perspective images corresponding to the target object based on the original image and the plurality of perspective information of the target object by using a target image generation model; performing three-dimensional reconstruction on the target object based on the plurality of perspective images to obtain an initial three-dimensional model of the target object; learning the association between the target object and the set marker information by using a text-to-image model according to the plurality of perspective images and the initial prompt word containing the set marker information; generating a plurality of scene images corresponding to the target object based on the plurality of scene prompt words containing the set marker information and the association between the target object and the set marker information learned by the text-to-image model; optimizing the initial three-dimensional model according to the plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object.

2. The method of claim 1, wherein, Based on the original image and the plurality of perspective information of the target object, the target image generation model is used to generate a plurality of perspective images corresponding to the target object, which comprises the steps of: inputting the original image and the plurality of perspective information into the information encoder in the target image generation model for vector embedding to obtain the semantic vectors corresponding to the original image and the plurality of perspective information respectively; inputting the semantic vectors corresponding to the original image and the plurality of perspective information as generation conditions into the diffusion generator in the target image generation model for denoising processing to obtain the feature vectors corresponding to the plurality of perspective information; inputting the feature vectors corresponding to the plurality of perspective information into the image decoder in the target image generation model for decoding processing respectively to obtain the plurality of perspective images corresponding to the target object.

3. The method of claim 2, wherein, It also comprises the steps of: model training on an initial image generation model according to a sample image and a plurality of sample perspective information to obtain a plurality of sample perspective images; calculating at least a pixel consistency loss function between the sample image and the plurality of sample perspective images, and updating the model parameters of the initial image generation model to obtain the target image generation model if the pixel consistency loss function does not satisfy a first preset loss condition.

4. The method of claim 3, wherein, The model training on the initial image generation model according to the sample image and the plurality of sample perspective information to obtain the plurality of sample perspective images comprises the steps of: inputting the plurality of sample perspective information into the text encoder in the initial image generation model for vector embedding to obtain the semantic vectors of the plurality of sample perspective information; inputting the sample image into the image encoder in the initial image generation model for feature extraction to obtain a first sample feature vector; adding an original sample noise signal to the first sample feature vector to obtain a second sample feature vector; inputting the semantic vectors of the plurality of sample perspective information and the second sample feature vector into the diffusion generator in the initial image generation model for noise prediction to obtain a plurality of predicted sample noise signals corresponding to the plurality of sample perspective information; subtracting the plurality of predicted sample noise signals from the second sample feature vector to obtain a plurality of third sample feature vectors; inputting the plurality of third sample feature vectors into an image decoder in the initial image generation model to obtain the plurality of sample view images.

5. The method of claim 3, wherein, calculating a pixel consistency loss function between the sample image and the plurality of sample view images, comprising: generating a mapping image of a first foreground segmentation region in the sample image under the plurality of sample view information according to an affine transformation matrix corresponding to the plurality of sample view information; extracting a second foreground segmentation region in the plurality of sample view images according to a pixel coordinate of the first foreground segmentation region in the sample image; calculating a pixel consistency loss function between the mapping image of the first foreground segmentation region under the plurality of sample view information and the second foreground segmentation region in the plurality of sample view images.

6. The method of claim 1, wherein, performing three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial three-dimensional model of the target object, comprising: performing neural radiance field-based three-dimensional reconstruction on the target object based on the plurality of view images to obtain an initial implicit three-dimensional representation model of the target object; inputting the original image into the initial implicit three-dimensional representation model to generate an image under a specified view to obtain a specified view image; calculating a depth loss function and / or a score distillation sampling (SDS) loss function between the specified view image and the original image; if the depth loss function and / or the score distillation sampling loss function does not satisfy a second preset loss condition, updating model parameters of the initial implicit three-dimensional representation model until the second preset loss condition is satisfied.

7. The method of claim 6, wherein, calculating a score distillation sampling (SDS) loss function between the specified view image and the original image, comprising: inputting the specified view image into an image encoder in the noise prediction model to extract a first initial feature vector; adding a first original noise signal to the first initial feature vector to obtain a first noise-added feature vector; inputting the first noise-added feature vector into a diffusion generator in the noise prediction model to predict noise to obtain a first predicted noise signal; calculating a mean square error loss function between the first original noise signal and the first predicted noise signal, the mean square error loss function comprising a gradient of the initial implicit three-dimensional representation model, a gradient of the noise prediction model, and noise contrast information; obtaining a sum of the gradient of the initial implicit three-dimensional representation model and the noise contrast information as the SDS loss function between the specified view image and the original image.

8. The method of claim 6, wherein, optimizing the initial three-dimensional model according to a plurality of scene images corresponding to the target object to obtain a target three-dimensional model of the target object, comprising: performing neural network-based three-dimensional reconstruction based on the plurality of scene images and the initial implicit three-dimensional representation model to obtain a target implicit three-dimensional representation model of the target object.

9. The method of claim 1, wherein, learning an association relationship between the target object and the set marker information using a text-to-image model according to the plurality of view images and an initial prompt word containing the set marker information, comprising: generate the initial prompt word in a prompt word format containing category information and label information, the initial prompt word including category information of the target object and the set label information; input the initial prompt word into a text encoder in the text-to-image model for vector embedding to obtain a semantic vector of the initial prompt word; input the multiple perspective images into an image encoder in the text-to-image model for feature extraction to obtain a second initial feature vector; add a second original noise signal to the second initial feature vector to obtain a second noise-added feature vector; input the semantic vector of the initial prompt word and the second noise-added feature vector into a diffusion generator in the text-to-image model for noise prediction to obtain a second predicted noise signal; if a loss function between the second original noise signal and the second predicted noise signal does not satisfy a third preset loss condition, update model parameters of the diffusion generator in the text-to-image model until the third preset loss condition is satisfied.

10. The method of claim 1, wherein, generate multiple scene images corresponding to the target object based on multiple scene prompt words containing the set label information and the learned association between the target object and the set label information by the text-to-image model, including: input the multiple scene prompt words containing the set label information into a text encoder in the text-to-image model for vector embedding to obtain semantic vectors corresponding to the multiple scene prompt words; input the semantic vectors corresponding to the multiple scene prompt words as a generation condition into a diffusion generator in the text-to-image model for denoising processing to obtain feature vectors corresponding to the multiple scene prompt words; input the multiple feature vectors into an image decoder of the text-to-image model for decoding processing to obtain the multiple scene images corresponding to the target object.

11. The method according to any one of claims 1 to 10, characterized in that, before optimizing the initial three-dimensional model based on the multiple scene images corresponding to the target object to obtain a target three-dimensional model of the target object, further comprising: for any scene image, input the any scene image into a segmentation module in an image inpainting model for image segmentation to segment the any scene image into multiple image blocks; input the multiple image blocks into a semantic recognition module in the image inpainting model for semantic recognition to obtain semantic information of the multiple image blocks; input the semantic information of the multiple image blocks into a detail inpainting module in the image inpainting model for semantic-based image detail inpainting to obtain multiple inpainted image blocks; splice the multiple inpainted image blocks to obtain an inpainted scene image.

12. An image generation method characterized by comprising: including: generate multiple perspective images corresponding to the target object based on an original image of the target object and multiple perspective information by using a target image generation model; perform three-dimensional reconstruction on the target object based on the multiple perspective images to obtain an initial three-dimensional model of the target object; learn an association between the target object and set label information according to the multiple perspective images and an initial prompt word containing the set label information by using a text-to-image model; According to a plurality of scene prompt words containing the setting mark information, an association relationship between the target object and the setting mark information learned by the text-to-image model is used to generate a plurality of scene images corresponding to the target object.

13. An electronic device, comprising: Comprising: a memory and a processor; the memory, configured to store a computer program; and the processor, coupled to the memory, configured to execute the computer program in the memory, so as to implement the steps in the method in any one of claims 1-11 and 12.

14. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is enabled to implement the steps in the method in any one of claims 1-11 and 12.

15. A computer program product, characterised in that, comprising computer programs / instructions, which, when executed by a processor, enable the processor to implement the steps in the method in any one of claims 1-11 and 12.

Images