A three-dimensional data generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a three-dimensional data generation method and device, electronic equipment and storage medium. The method comprises: obtaining a first image of a target object and a description text corresponding to the first image; obtaining a noise image; wherein the noise image comprises a rendered image with added noise; the rendered image comprises an image rendered by a neural radiance field at a preset viewing angle; cross-attention fusion is performed on the features of the first image, the features of the description text and the features of the noise image by a diffusion network to generate a second image; wherein the second image comprises an image of the target object at the preset viewing angle; a score distillation loss is determined according to the second image, the noise image and the added noise, and the neural radiance field is adjusted according to the score distillation loss; and a three-dimensional model of the target object is determined according to the adjusted neural radiance field. High-quality three-dimensional data can be quickly and efficiently generated.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a three-dimensional data generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] The traditional three-dimensional (3D) content creation process is complicated and requires a large amount of manpower and material resources. How to quickly and efficiently generate high-quality three-dimensional data is a technical problem to be solved. SUMMARY

[0003] Embodiments of the present disclosure provide a three-dimensional data generation method and device, electronic equipment and storage medium, which can quickly and efficiently generate high-quality three-dimensional data.

[0004] In a first aspect, the embodiments of the present disclosure provide a three-dimensional data generation method, comprising:

[0005] obtaining a first image of a target object and a description text corresponding to the first image;

[0006] obtaining a noise image; wherein the noise image includes a rendered image with added noise; the rendered image includes an image rendered by a neural radiance field at a preset viewing angle;

[0007] crossing attention fusion of features of the first image, features of the description text and features of the noise image through a diffusion network to generate a second image; wherein the second image includes an image of the target object at the preset viewing angle;

[0008] determining a score distillation loss according to the second image, the noise image and the added noise, and adjusting the neural radiance field according to the score distillation loss;

[0009] determining a three-dimensional model of the target object according to the adjusted neural radiance field.

[0010] In a second aspect, the embodiments of the present disclosure also provide a three-dimensional data generation device, comprising:

[0011] a first obtaining module configured to obtain a first image of a target object and a description text corresponding to the first image;

[0012] a second obtaining module configured to obtain a noise image; wherein the noise image includes a rendered image with added noise; the rendered image includes an image rendered by a neural radiance field at a preset viewing angle;

[0013] A diffusion module is used to perform cross-attention fusion of features of the first image, features of the descriptive text, and features of the noisy image through a diffusion network to generate a second image; wherein, the second image includes an image of the target object from the preset viewpoint;

[0014] A three-dimensional representation module is used to determine fractional distillation loss based on the second image, the noise image, and the added noise, and to adjust the neural radiation field based on the fractional distillation loss;

[0015] The model generation module is used to determine the three-dimensional model of the target object based on the adjusted neural radiation field.

[0016] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs.

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating three-dimensional data as described in any embodiment of this disclosure.

[0020] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for generating three-dimensional data as described in any of the embodiments of this disclosure.

[0021] The technical solution of this disclosure includes: acquiring a first image of a target object and descriptive text corresponding to the first image; acquiring a noisy image; wherein the noisy image includes a rendered image with added noise; the rendered image includes an image rendered by a neural radiation field under a preset viewpoint; using a diffusion network, cross-attention fusion is performed on the features of the first image, the features of the descriptive text, and the features of the noisy image to generate a second image; wherein the second image includes an image of the target object under a preset viewpoint; determining a fractional distillation loss based on the second image, the noisy image, and the added noise, and adjusting the neural radiation field based on the fractional distillation loss; and determining a three-dimensional model of the target object based on the adjusted neural radiation field.

[0022] By cross-attention fusion of the features of the first image, the features of the description text and the features of the noise image based on the diffusion network, the second image can be generated based on the noise image with the first image and the description text as the control condition. By determining the rendering image under the preset view angle based on the neural radiance field, generating the noise image for inputting into the diffusion network according to the rendering image in a manner of adding noise, and constructing the neural radiance field through the fractional distillation loss, the three-dimensional model expressed based on the neural radiance field can have the same feature distribution as the second image generated based on the first image and the description text. Therefore, high-quality three-dimensional data can be quickly and efficiently generated according to the first image and the description text. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and other features, aspects, and advantages of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or like reference numerals refer to the same or like elements. It is to be understood that the drawings are schematic, and the proportions of the elements and features do not necessarily bear a relationship of scale.

[0024] Figure 1 A flowchart of a three-dimensional data generation method provided by an embodiment of the present disclosure;

[0025] Figure 2 A framework diagram of a three-dimensional data generation method provided by an embodiment of the present disclosure;

[0026] Figure 3 A diagram of feature cross-attention fusion in a three-dimensional data generation method provided by an embodiment of the present disclosure;

[0027] Figure 4 A flowchart of a three-dimensional data generation method provided by an embodiment of the present disclosure;

[0028] Figure 5 A diagram of a third image in a three-dimensional data generation method provided by an embodiment of the present disclosure;

[0029] Figure 6 A flowchart of a three-dimensional data generation method provided by an embodiment of the present disclosure;

[0030] Figure 7 A structural diagram of a three-dimensional data generation device provided by an embodiment of the present disclosure;

[0031] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather should be construed to encompass all modifications equivalent in meaning and scope. It is understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0033] It should be understood that each step described in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related definitions are given below in the description of the embodiments.

[0035] It should be noted that the terms "first", "second", and the like used in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the terms "one", "multiple" used in the present disclosure are illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0038] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0039] Figure 1 A flowchart of a three-dimensional data generation method provided by an embodiment of the present disclosure is shown. The embodiment of the present disclosure is applicable to the case of generating a similar three-dimensional model according to a picture and text. The method can be performed by a three-dimensional data generation device, which can be implemented in the form of software and / or hardware, and can be configured in an electronic device, such as a computer.

[0040] As Figure 1As shown, the three-dimensional data generation method provided by the embodiment can include:

[0041] S110, acquire a first image of a target object and a description text corresponding to the first image.

[0042] In the embodiment of the present disclosure, the target object can be an object or a living object. The first image includes an image containing the target object, the description text can be considered as a text describing the target object in the first image, and the first image and the description text have content consistency. The first image and the description text can be used as a control condition for generating a three-dimensional model of the target object, so that the generated three-dimensional model has similarity with the first image and the description text.

[0043] The first image and the description text can be obtained by user input, or can be obtained by reading a preset storage space, and the like, which is not exhaustive.

[0044] S120, acquire a noise image; wherein the noise image includes a rendered image with added noise; the rendered image includes an image rendered by a neural radiance field at a preset view angle.

[0045] In the embodiment of the present disclosure, the neural radiance field (NeRF) can be randomly initialized, and the camera parameters of the preset view angle can be sampled in the spherical coordinates, and the point light sources around the camera can be sampled. Then, the NeRF field can be rendered at the preset view angle, thereby obtaining a rendered image.

[0046] After obtaining the rendered image of the NeRF at the preset view angle, noise (such as Gaussian noise) can be added to the rendered image to obtain a noise image. The preset view angle can include at least one view angle, and correspondingly, the rendered image can include at least one image, and the noise image can also include at least one image.

[0047] Exemplarily, Figure 2 A framework diagram of a three-dimensional data generation method provided by the embodiment of the present disclosure is shown. Referring to Figure 2 The NeRF field is rendered at the preset view angle to obtain a rendered image a, and noise b is added to the rendered image to obtain a noise image c.

[0048] S130, cross-attention fusion is performed on the features of the first image, the features of the description text, and the features of the noise image by a diffusion network to generate a second image; wherein the second image includes an image of the target object at a preset view angle.

[0049] In the embodiments of the present disclosure, the diffusion network can include a two-dimensional diffusion network, and the diffusion network can be a network model that has been constructed (i.e., the parameters in the diffusion network in the embodiments are fixed). Referring again to Figure 2 , the diffusion network can perform inverse diffusion on the input noise image c to generate a second image e. Since the noise image c is generated based on the rendered image a at the preset view angle, the second image e also belongs to the image at the preset view angle.

[0050] Figure 2 In the embodiments of the present disclosure, in the process of generating the second image by the diffusion network, the extracted features of the noise image c can be cross-attention fused with the features of the first image d and the features of the description text t, so that the generated second image has similarity with the first image and the description text.

[0051] An exemplary Figure 3 schematic diagram of cross-attention fusion of features in a three-dimensional data generation method provided by the embodiments of the present disclosure. Referring to Figure 3 In some optional implementations, cross-attention fusion of the features of the first image, the features of the description text, and the features of the noise image by the diffusion network can include: performing feature processing on the noise image by each denoising sub-network in the diffusion network; performing feature processing on the first image by a mirror network having the same structure as the diffusion network; performing feature extraction on the description text by a text feature extraction network; and cross-attention fusing, by a cross-attention sub-network in the diffusion network, the features of the noise image processed by the denoising sub-network in the diffusion network, the features of the first image processed by the corresponding denoising sub-network in the mirror network, and the features of the description text.

[0052] Referring to Figure 3 , the diffusion network can include a plurality of denoising sub-networks (for example, a U-Net network in Figure 3 ), and the denoising sub-networks can perform feature processing on the input noise image. Figure 3In some optional implementation manners, the feature processing on the first image through the mirror network with the same structure as the diffusion network can include: performing feature extraction on the first image through an image encoder to obtain a first feature map, and then performing feature processing on the first feature map through the mirror network with the same structure as the diffusion network. It should be noted that the mirror network and the diffusion network having the same structure can mean that both of them include the same number of denoising sub-networks.

[0053] Figure 3 In some optional implementation manners, the diffusion network can further include a cross-attention sub-network; and the features of the processed noise image in each denoising sub-network of the diffusion network can be cross-attention fused with the features of the processed first image in the corresponding denoising sub-network of the mirror network and the features of the description text through the cross-attention sub-network. That is, the features of the processed noise image in the i th denoising sub-network of the diffusion network can be cross-attention fused with the features of the processed first image in the i th denoising sub-network of the mirror network and the features of the description text.

[0054] In some optional implementation manners, the cross-attention fusion of the features of the processed noise image in the denoising sub-network of the diffusion network with the features of the processed first image in the corresponding denoising sub-network of the mirror network and the features of the description text can include: cross-attention fusion of the features of the processed noise image in at least part of the denoising sub-networks of the diffusion network with the features of the processed first image in the corresponding denoising sub-networks of the mirror network and the features of the description text. In the diffusion network, a cross-attention sub-network can be correspondingly deployed for each denoising sub-network that needs to be cross-attention fused.

[0055] In some optional implementation manners, the cross-attention fusion of the features of the processed noise image in the denoising sub-network of the diffusion network with the features of the processed first image in the corresponding denoising sub-network of the mirror network and the features of the description text can include: cross-attention fusion of the features of the processed noise image in at least part of the denoising sub-networks of the diffusion network with the features of the processed first image in the corresponding denoising sub-networks of the mirror network and the features of the description text. In the diffusion network, a cross-attention sub-network can be correspondingly deployed for each denoising sub-network that needs to be cross-attention fused.

[0056] In the embodiments of the present disclosure, the first image is processed by using a mirror network consistent with the diffusion network, and the intermediate features of the noise image processed by at least part of the denoising sub-network in the mirror network are fused into the corresponding denoising sub-network in the diffusion network through the cross-attention sub-network, and meanwhile the cross-attention sub-network can also fuse the features of the description text into the corresponding denoising sub-network in the diffusion network, so that the generated second image has high similarity with the first image and the description text.

[0057] In addition, at least part of the denoising sub-networks that need to be fused by cross-attention and at least one network layer in the denoising sub-networks that need to be fused by cross-attention can be set according to the actual application scene, so as to save the network calculation amount and improve the second image generation efficiency to a certain extent while ensuring the similarity between the second image and the first image and the description text.

[0058] S140, determining a score distillation loss according to the second image, the noise image and the added noise, and adjusting the neural radiance field according to the score distillation loss.

[0059] Again referring to Figure 2 By comparing the second image e and the noise image c, the added noise f predicted by the diffusion network can be determined. According to the predicted added noise f and the actual added noise b, the score distillation loss (Score Distillation Sampling, SDS) can be determined. Wherein, the closer the feature distribution of the picture rendered based on NeRF is to the second image, the closer the noise predicted by the diffusion network is to the actual added noise, and therefore the parameters in NeRF can be adjusted based on the score distillation loss, so that the rendering result of NeRF has similarity with the feature distribution of the second image. That is, the loss based on the two-dimensional image can be used to construct the implicit expression of the three-dimensional model.

[0060] S150, determining the three-dimensional model of the target object according to the adjusted neural radiance field.

[0061] In the embodiments of the present disclosure, the three-dimensional model can include a three-dimensional mesh model. The initial three-dimensional model can be extracted based on the constructed NeRF based on the existing algorithm. For example, the initial three-dimensional model can be extracted based on the constructed NeRF based on the Deep Marching Tetrahedra (DMTet) algorithm. In addition, a high-resolution three-dimensional model can also be generated based on the initial three-dimensional model based on the existing algorithm. The generated three-dimensional model of the target object can have high similarity with the first image and the description text. Compared with the generation of the traditional three-dimensional model, the technical solution of the embodiments of the present disclosure can quickly and efficiently generate a high-quality three-dimensional model according to the first image and the description text.

[0062] The technical scheme of the embodiment of the present disclosure is as follows: a first image of a target object and description text corresponding to the first image are obtained; a noise image is obtained; the noise image includes a rendered image to which noise is added; the rendered image includes an image rendered by a neural radiance field at a preset view angle; features of the first image, features of the description text, and features of the noise image are cross-attention fused by a diffusion network to generate a second image; the second image includes an image of the target object at the preset view angle; a score distillation loss is determined according to the second image, the noise image, and the added noise, and the neural radiance field is adjusted according to the score distillation loss; and a three-dimensional model of the target object is determined according to the adjusted neural radiance field.

[0063] By cross-attention fusing the features of the first image, the features of the description text, and the features of the noise image based on the diffusion network, the second image can be generated based on the noise image with the first image and the description text as control conditions. By determining the rendered image at the preset view angle based on the neural radiance field, generating the noise image for inputting into the diffusion network from the rendered image in the manner of adding noise, and constructing the neural radiance field through the score distillation loss, the three-dimensional model expressed based on the neural radiance field can have the same feature distribution as the second image generated based on the first image and the description text. Therefore, high-quality three-dimensional data can be quickly and efficiently generated according to the first image and the description text.

[0064] The embodiment of the present disclosure can be combined with each optional scheme in the three-dimensional data generation method provided in the above embodiment. The three-dimensional data generation method provided in this embodiment is described in detail in the case where the target object includes a target human object. By additionally adding a third image in a preset posture as a control condition for generating the second image, the generated three-dimensional model can also be in the preset posture, thereby facilitating further processing of the three-dimensional model, such as bone binding and the like.

[0065] Figure 4 A flowchart of a three-dimensional data generation method provided by an embodiment of the present disclosure is shown in FIG. 4. As shown in FIG. 4, the three-dimensional data generation method provided by the embodiment can include the following steps. Figure 4

[0066] S410, a first image of a target human object and description text corresponding to the first image are obtained.

[0067] S420, a noise image is obtained; the noise image includes a rendered image to which noise is added; the rendered image includes an image rendered by a neural radiance field at a preset view angle.

[0068] ​S430, acquire a third image of the preset three-dimensional model under a preset view angle; wherein, the preset three-dimensional model comprises a preset human body model in a preset posture.

[0069] In the embodiment, the preset three-dimensional model can comprise a three-dimensional mesh model. Wherein, the preset three-dimensional model can be generated based on an existing neural network conditioned on posture control parameters. For example, a standard model can be deformed to generate a preset human body model in a preset posture according to posture control parameters of each preset region through a Skinned Multi-Person Linear eXpressive (SMPL-X) model.

[0070] Wherein, the preset three-dimensional model can be rendered from each preset view angle to obtain each third image; wherein, each third image can present a full-surface view of the preset human body. For example, Figure 5 A schematic diagram of the third image in the three-dimensional data generation method provided by the embodiment of the present disclosure. Referring to Figure 5 The preset three-dimensional model is in a natural standing posture, and the third image of the preset three-dimensional model under the front view angle and the back view angle can be acquired.

[0071] S440, generate a second image through a diffusion network based on the features of the first image, the features of the description text, the features of the noise image, and the features of the third image; wherein, the second image comprises an image of a target human body object in a preset posture under a preset view angle.

[0072] In the embodiment, the diffusion network can cross-attention fuse the features of the first image, the features of the description text, and the features of the noise image, and can also extract the features of the third image and introduce the features of the third image into the denoising sub-network of the diffusion network. Thus, the generated second image can maintain similarity with the first image and the text description, and can also present the same preset posture as the preset human body model in the third image, so as to adjust the posture of the three-dimensional model for presentation.

[0073] S450, determine a score distillation loss according to the second image, the noise image, and the added noise, and adjust the neural radiance field according to the score distillation loss.

[0074] S460, determine a three-dimensional model of the target human body object according to the adjusted neural radiance field.

[0075] In some optional implementations, after determining the three-dimensional model of the target human body object, the method can further comprise: binding the three-dimensional model of the target human body object with a skeletal model in a preset posture; wherein, the three-dimensional model of the bound skeletal model is driven based on posture control parameters.

[0076] In the optional implementation, the preset pose can be a pose presented by the existing skeleton model. Since the three-dimensional model is the same as the pose presented by the skeleton model, the standard key points of the three-dimensional model and the skeleton model can be aligned very conveniently. The three-dimensional model and the skeleton model can be bound by using the existing binding tool, and then the generation of the dynamic three-dimensional model can be realized by driving the skeleton model through the pose control parameter.

[0077] The technical solutions of the embodiments of the present disclosure are described in detail in the case where the target object includes a target human object. By adding the third image in the preset pose as the control condition for generating the second image, the generated three-dimensional model can also be in the preset pose, thereby facilitating the further processing of the three-dimensional model, such as the bone binding and the like. The three-dimensional data generation method provided by the embodiments of the present disclosure belongs to the same disclosure concept as the three-dimensional data generation method provided by the above embodiments, and the technical details not described in detail in the present embodiment can be referred to the above embodiments, and the same technical features have the same beneficial effects in the present embodiment and the above embodiments.

[0078] The three-dimensional data generation method provided by the embodiments of the present disclosure can be combined with the optional schemes in the three-dimensional data generation method provided by the above embodiments. The three-dimensional data generation method provided by the present embodiment is described in detail in the application scenario. By detecting and segmenting the human objects and scene objects in the target video to obtain the segmented image, the segmented image can be used as the control condition for generating the three-dimensional model, so that the generated three-dimensional model has similarity with the human objects and scene objects in the target video. Based on this, the three-dimensional scene similar to the target video can be built and rendered.

[0079] Figure 6 A flowchart of a three-dimensional data generation method provided by the embodiments of the present disclosure is shown in FIG. 6. As shown in FIG. 6, the three-dimensional data generation method provided by the present embodiment can include the following steps. Figure 6

[0080] S610, detecting the objects in the target video to determine the video frame containing the target object.

[0081] In the present embodiment, the existing image processing algorithm can be used to detect and track the objects (such as scene objects and human objects) in each frame of the target video. When a new object is detected, the detected object instance can be saved, and the video frame containing the object instance can be recorded. At least one of the detected objects can be determined as the target object. Thus, the video frame containing the target object in the target video can be determined.

[0082] S620, determining the target video frame from the video frame containing the target object. ​

[0083] The target video frame can be determined from each video containing the target object based on a preset selection strategy. For example, the video frame in which the target object region has the highest proportion in each video frame can be determined as the target video frame. For another example, the video frame in which the target object has the highest definition in each video frame can be determined as the target video frame. In addition, other selection strategies can also be applied, which will not be listed here.

[0084] S630, segmenting the target video frame to obtain a first image of the target object.

[0085] In this embodiment, the target object region in the target video frame can be segmented based on an existing image segmentation algorithm, and a first image of the target object can be obtained.

[0086] S640, generating a description text corresponding to the first image based on the first image of the target object through an image description network.

[0087] In this embodiment, the image description (image caption) network can include an existing image annotation generation network, for example, it can include a Transformer-based model, such as Bidirectional Encoder Representations from Transformers (BERT) and the like. The first image can be input into the image description network to make it output the corresponding description text. Thus, the first image and the description text of the target object in the target video frame can be obtained.

[0088] S650, obtaining a noise image; wherein the noise image includes a rendered image with added noise; the rendered image includes an image rendered by the neural radiance field at a preset view angle.

[0089] S660, cross-attention fusion of the features of the first image, the features of the description text, and the features of the noise image through a diffusion network to generate a second image; wherein the second image includes an image of the target object at a preset view angle.

[0090] In the case where the target object includes a target human object, the method can further include: obtaining a third image of a preset three-dimensional model at a preset view angle; wherein the preset three-dimensional model includes a preset human model in a preset posture.

[0091] Correspondingly, the feature of the first image, the feature of the description text and the feature of the noise image are cross-attention fused through the diffusion network to generate the second image, which can include: generating the second image based on the feature of the first image, the feature of the description text, the feature of the noise image and the feature of the third image through the diffusion network; wherein the second image includes an image of the target human object in a preset pose under a preset visual angle.

[0092] S670, determine a score distillation loss according to the second image, the noise image and the added noise, and adjust the neural radiance field according to the score distillation loss.

[0093] S680, determine the three-dimensional model of the target object according to the adjusted neural radiance field.

[0094] Wherein, the corresponding neural radiance field can be adjusted for each object to determine the three-dimensional model corresponding to each target object.

[0095] S690, generate a three-dimensional scene corresponding to the target video according to the three-dimensional model of the target object.

[0096] Wherein, the three-dimensional model corresponding to the target object in the target video can be used to build a three-dimensional scene according to the spatial position, visual angle direction and the like of the target object in the target video, so as to generate a three-dimensional scene with high similarity to the target video. Thus, the rapid construction of a corresponding three-dimensional scene based on a two-dimensional video can be realized, which has wide application prospects and can meet the needs of users for three-dimensional scene construction.

[0097] In addition, the preset three-dimensional model material can also be placed in the constructed three-dimensional scene to enrich the three-dimensional scene. The three-dimensional scene can also be rendered differently from the visual angle, light and the like of the target video to further enrich the presentation of the three-dimensional scene.

[0098] In some optional implementations, in the case where the target object includes a target human object, the three-dimensional data generation method can further include: performing motion estimation on the target human object in the target video to determine a sequence of pose control parameters; and driving the three-dimensional model of the target human object according to the sequence of pose control parameters.

[0099] Wherein, the target human object in the video frame corresponding to the target human object can be motion estimated based on an existing human pose estimation model (such as a dense pose model) to determine the joint pose parameters (such as position, angle and the like) of the target human object in each frame, so as to obtain the sequence of pose control parameters of each joint.

[0100] After the posture control parameter sequence is determined, the posture control sequence of each joint can be loaded into the corresponding three-dimensional model, so that the driving of the three-dimensional model of the target human object can be realized. The movement of the three-dimensional model of the target human object in the three-dimensional scene can be realized, which has high similarity with the target video. Therefore, the dynamic scene can be built, and the user experience can be improved.

[0101] The technical solutions of the embodiments of the present disclosure are described in detail. By detecting and segmenting the human object and the scene object in the target video, a segmentation image is obtained, which can be used as a control condition for generating a three-dimensional model, so that the generated three-dimensional model has similarity with the human object and the scene object in the target video. Based on this, the construction and rendering of the three-dimensional scene similar to the target video can be realized. The three-dimensional data generation method provided by the embodiments of the present disclosure belongs to the same disclosure concept as the three-dimensional data generation method provided by the above embodiments. The technical details not described in detail in the present embodiment can be referred to the above embodiments, and the same technical features have the same beneficial effects in the present embodiment and the above embodiments.

[0102] Figure 7 A structural schematic diagram of a three-dimensional data generation device provided by the embodiments of the present disclosure is shown. The three-dimensional data generation device provided by the present embodiment is suitable for the case of generating a similar three-dimensional model according to a picture and a text.

[0103] As Figure 7 shown, the three-dimensional data generation device provided by the embodiments of the present disclosure can include:

[0104] The first acquisition module 710 is configured to acquire a first image of a target object and a description text corresponding to the first image.

[0105] The second acquisition module 720 is configured to acquire a noise image; wherein the noise image includes a rendered image with added noise; and the rendered image includes an image rendered by a neural radiance field under a preset view angle.

[0106] The diffusion module 730 is configured to cross-attention fuse the features of the first image, the features of the description text and the features of the noise image through a diffusion network to generate a second image; wherein the second image includes an image of the target object under the preset view angle.

[0107] The three-dimensional representation module 740 is configured to determine a score distillation loss according to the second image, the noise image and the added noise, and adjust the neural radiance field according to the score distillation loss.

[0108] The model generation module 750 is configured to determine a three-dimensional model of the target object according to the adjusted neural radiance field.

[0109] In some optional implementations, the diffusion module can be configured to:

[0110] perform feature processing on the noisy image by each denoising subnetwork in the diffusion network;

[0111] perform feature processing on the first image by a mirror network having the same structure as the diffusion network;

[0112] perform feature extraction on the description text by a text feature extraction network;

[0113] perform cross-attention fusion on the features of the noisy image processed by the denoising subnetworks in the diffusion network, the features of the first image processed by the corresponding denoising subnetworks in the mirror network, and the features of the description text by a cross-attention subnetwork in the diffusion network.

[0114] In some optional implementations, the diffusion module can be configured to:

[0115] perform cross-attention fusion on the features of at least part of the noisy images processed by the denoising subnetworks in the diffusion network, the features of the first image processed by the corresponding denoising subnetworks in the mirror network, and the features of the description text.

[0116] In some optional implementations, the diffusion module can be configured to:

[0117] perform cross-attention fusion on the features of the noisy image processed by at least part of the network layers in the denoising subnetwork, the features of the first image processed by the corresponding network layers in the corresponding denoising subnetwork, and the features of the description text.

[0118] In some optional implementations, when the target object includes a target human object, the three-dimensional data generation apparatus can further include:

[0119] a third acquisition module configured to acquire a third image of a preset three-dimensional model under a preset perspective; the preset three-dimensional model includes a preset human model in a preset posture;

[0120] Correspondingly, the diffusion module can be configured to:

[0121] generate a second image based on the features of the first image, the features of the description text, the features of the noisy image, and the features of the third image by the diffusion network; the second image includes an image of the target human object in the preset posture under the preset perspective.

[0122] In some optional implementations, the three-dimensional data generation apparatus can further include:

[0123] The model driving module is configured to bind the three-dimensional model of the target human object to a skeleton model in a preset pose after determining the three-dimensional model of the target human object.

[0124] The three-dimensional model of the bound skeleton model is driven based on the pose control parameter.

[0125] In some optional implementations, the first obtaining module can be configured to:

[0126] detect an object in the target video, and determine a video frame containing the target object;

[0127] determine a target video frame from the video frame containing the target object;

[0128] segment the target video frame to obtain a first image of the target object.

[0129] In some optional implementations, the first obtaining module can be further configured to:

[0130] generate a description text corresponding to the first image based on the first image of the target object through an image description network.

[0131] In some optional implementations, the model driving module can be configured to:

[0132] perform motion estimation on the target human object in the target video to determine a sequence of pose control parameters;

[0133] drive the three-dimensional model of the target human object according to the sequence of pose control parameters.

[0134] In some optional implementations, the model generating module can be further configured to:

[0135] generate a three-dimensional scene corresponding to the target video according to the three-dimensional model of the target object.

[0136] The three-dimensional data generation apparatus provided in the embodiments of the present disclosure can perform the three-dimensional data generation method provided in any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of performing the method.

[0137] It should be noted that each unit and module included in the apparatus is only divided according to the function logic, and is not limited to the above division, as long as the corresponding function can be implemented; in addition, the specific names of each functional unit are only for convenient distinction, and do not limit the protection scope of the embodiments of the present disclosure.

[0138] Reference will be made to the following description of the embodiments of the present disclosure Figure 8 which shows an electronic device (for example, a mobile phone) suitable for implementing the embodiments of the present disclosure. Figure 8The diagram below shows the structure of the terminal device or server 800. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0139] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0140] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0141] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined in the three-dimensional data generation method of embodiments of this disclosure.

[0142] The electronic device provided by the embodiments of the present disclosure and the method for generating three-dimensional data provided by the above embodiments belong to the same disclosure concept, and the technical details not described in detail in the embodiments can be referred to the above embodiments, and the embodiments have the same beneficial effects as the above embodiments.

[0143] The embodiments of the present disclosure provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the method for generating three-dimensional data provided by the above embodiments.

[0144] It should be noted that the computer readable medium of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0145] In some embodiments, the client, server, or both can communicate using any known or future developed network protocols, such as the Hyper Text Transfer Protocol (HTTP), and can be interconnected with any form or medium of digital data communication (for example, a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet, and peer-to-peer networks (for example, ad hoc peer-to-peer networks), as well as any current or future developed network.

[0146] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and can be accessed via the electronic device.

[0147] The computer-readable medium described above carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0148] Obtain a first image of a target object and a description text corresponding to the first image, and obtain a noise image; wherein the noise image includes a rendered image with added noise; the rendered image includes an image rendered by a neural radiance field at a preset viewing angle; cross-attention fuse features of the first image, features of the description text, and features of the noise image through a diffusion network to generate a second image; wherein the second image includes an image of the target object at the preset viewing angle; determine a score distillation loss according to the second image, the noise image, and the added noise, and adjust the neural radiance field according to the score distillation loss; and determine a three-dimensional model of the target object according to the adjusted neural radiance field.

[0149] Computer program code for carrying out operations of the present disclosure can be written in any one or more of a number of programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network ("LAN") or a wide area network ("WAN"), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a computer system, includes electronic components such as a computer, network, and / or other platforms for operations described herein.

[0150] The computer program product of the present disclosure can be a computer program product, which can be implemented as a tangible computer readable storage medium or computer readable non-transitory medium having instructions recorded thereon for one or more programmable processors to execute. The computer program product can include instructions for causing a processor or processors to perform functions and / or operations described herein. The instructions can include any suitable type of code and / or computer instructions.

[0151] The units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the names of the units, modules do not constitute a limitation on the units, modules themselves.

[0152] The functions described in the above description can be performed at least in part by one or more hardware logic components. For example, non-limiting examples of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] According to one or more embodiments of the present disclosure, a method for generating three-dimensional data is provided, the method comprising:

[0155] obtaining a first image of a target object and a description text corresponding to the first image;

[0156] obtaining a noise image; wherein the noise image comprises a rendered image with added noise; the rendered image comprises an image rendered by a neural radiance field at a preset view angle;

[0157] cross-attention fusing, by a diffusion network, features of the first image, features of the description text, and features of the noise image, to generate a second image; wherein the second image comprises an image of the target object at the preset view angle;

[0158] determining a score distillation loss according to the second image, the noise image, and the added noise, and adjusting the neural radiance field according to the score distillation loss;

[0159] determining a three-dimensional model of the target object according to the adjusted neural radiance field.

[0160] According to one or more embodiments of the present disclosure, a method for generating three-dimensional data is provided, further comprising:

[0161] In some optional implementations, the cross-attention fusing, by a diffusion network, of the features of the first image, the features of the description text, and the features of the noise image comprises:

[0162] processing, by each denoising sub-network in the diffusion network, features of the noise image;

[0163] The first image is processed by a feature processing network which is the same as the diffusion network structure;

[0164] The description text is processed by a text feature extraction network;

[0165] The features of the noise image processed by the denoising sub-network in the diffusion network are cross-attention fused with the features of the first image processed by the corresponding denoising sub-network in the mirror network and the features of the description text by a cross-attention sub-network in the diffusion network.

[0166] According to one or more embodiments of the present disclosure, a method for generating three-dimensional data is provided, further comprising:

[0167] In some optional implementations, the cross-attention fusion of the features of the noise image processed by the denoising sub-network in the diffusion network, the features of the first image processed by the corresponding denoising sub-network in the mirror network, and the features of the description text comprises:

[0168] The features of the noise image processed by at least part of the denoising sub-network in the diffusion network are cross-attention fused with the features of the first image processed by the corresponding denoising sub-network in the mirror network and the features of the description text.

[0169] According to one or more embodiments of the present disclosure, a method for generating three-dimensional data is provided, further comprising:

[0170] In some optional implementations, the cross-attention fusion of the features of the noise image processed by the denoising sub-network in the diffusion network, the features of the first image processed by the corresponding denoising sub-network in the mirror network, and the features of the description text comprises:

[0171] The features of the noise image processed by at least part of the denoising sub-network in the diffusion network are cross-attention fused with the features of the first image processed by the corresponding denoising sub-network in the mirror network and the features of the description text.

[0172] According to one or more embodiments of the present disclosure, a method for generating three-dimensional data is provided, further comprising:

[0173] In some optional implementations, in the case that the target object comprises a target human object, the method further comprises:

[0174] A third image of a preset three-dimensional model under the preset perspective is acquired; wherein the preset three-dimensional model comprises a preset human model in a preset posture;

[0175] Correspondingly, the diffusion network is used to cross-attention fuse the features of the first image, the features of the description text, and the features of the noise image to generate a second image, including:

[0176] The diffusion network is used to generate a second image based on the features of the first image, the features of the description text, the features of the noise image, and the features of the third image, wherein the second image includes an image of the target human object in the preset posture under the preset visual angle.

[0177] According to one or more embodiments of the present disclosure, a three-dimensional data generation method is provided, further comprising:

[0178] In some optional implementations, after determining the three-dimensional model of the target human object, the method further comprises:

[0179] Binding the three-dimensional model of the target human object with the skeletal model in the preset posture;

[0180] The three-dimensional model of the skeletal model is driven based on a posture control parameter.

[0181] According to one or more embodiments of the present disclosure, a three-dimensional data generation method is provided, further comprising:

[0182] In some optional implementations, the first image of the target object is obtained by:

[0183] Detecting the object in the target video to determine a video frame containing the target object;

[0184] Determining a target video frame from the video frame containing the target object;

[0185] Segmenting the target video frame to obtain the first image of the target object.

[0186] According to one or more embodiments of the present disclosure, a three-dimensional data generation method is provided, further comprising:

[0187] In some optional implementations, the description text corresponding to the first image is obtained by:

[0188] Using an image description network to generate the description text corresponding to the first image based on the first image of the target object.

[0189] According to one or more embodiments of the present disclosure, a three-dimensional data generation method is provided, further comprising:

[0190] In some optional implementations, in the case where the target object includes a target human object, the method further comprises:

[0191] performing motion estimation on the target human object in the target video to determine a sequence of pose control parameters;

[0192] driving a three-dimensional model of the target human object according to the sequence of pose control parameters.

[0193] According to one or more embodiments of the present disclosure, a three-dimensional data generation method is provided, further comprising:

[0194] In some optional implementations, after the three-dimensional model of the target object is determined, further comprising:

[0195] generating a three-dimensional scene corresponding to the target video according to the three-dimensional model of the target object.

[0196] According to one or more embodiments of the present disclosure, a three-dimensional data generation apparatus is provided, comprising:

[0197] a first acquisition module configured to acquire a first image of a target object and a description text corresponding to the first image;

[0198] a second acquisition module configured to acquire a noise image; wherein the noise image comprises a rendered image with added noise; and the rendered image comprises an image rendered by a neural radiance field under a preset view angle;

[0199] a diffusion module configured to cross-attentionally fuse features of the first image, features of the description text, and features of the noise image through a diffusion network to generate a second image; wherein the second image comprises an image of the target object under the preset view angle;

[0200] a three-dimensional representation module configured to determine a score distillation loss according to the second image, the noise image, and the added noise, and adjust the neural radiance field according to the score distillation loss;

[0201] a model generation module configured to determine a three-dimensional model of the target object according to the adjusted neural radiance field.

[0202] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology employed. Those skilled in the art should understand that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also encompass other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the above disclosed concept. For example, the above features can be replaced with technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.

[0203] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the disclosure. Certain of the operations described in the discussion are combinable into a single operation, and certain operations can be separated into several operations. In some embodiments, the operations described in the discussion can be performed in an order different than presented in the discussion. In some embodiments, the operations described in the discussion can be performed concurrently. Also, while several specific implementation details are discussed in the discussion, these should not be interpreted as limiting the scope of the disclosure. Rather, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0204] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method of generating three-dimensional data, characterized by, The method comprises the following steps: obtaining a first image of a target object and a description text corresponding to the first image; obtaining a noise image; wherein the noise image comprises a rendered image with added noise; the rendered image comprises an image rendered by a neural radiance field under a preset view angle; cross-attention fusion of features of the first image, features of the description text and features of the noise image is performed by a diffusion network to generate a second image; wherein the second image comprises an image of the target object under the preset view angle; a score distillation loss is determined according to the second image, the noise image and the added noise, and the neural radiance field is adjusted according to the score distillation loss; a three-dimensional model of the target object is determined according to the adjusted neural radiance field.

2. The method of claim 1, wherein, The cross-attention fusion of the features of the first image, the features of the description text and the features of the noise image by the diffusion network comprises: feature processing of the noise image is performed by each denoising subnetwork in the diffusion network; feature processing of the first image is performed by a mirror network with the same structure as the diffusion network; feature extraction of the description text is performed by a text feature extraction network; cross-attention fusion of the features of the noise image processed by the denoising subnetwork in the diffusion network, the features of the first image processed by the corresponding denoising subnetwork in the mirror network and the features of the description text is performed by a cross-attention subnetwork in the diffusion network.

3. The method of claim 2, wherein, The cross-attention fusion of the features of the noise image processed by the denoising subnetwork in the diffusion network, the features of the first image processed by the corresponding denoising subnetwork in the mirror network and the features of the description text comprises: cross-attention fusion of the features of the noise image processed by at least part of the denoising subnetwork in the diffusion network, the features of the first image processed by the corresponding denoising subnetwork in the mirror network and the features of the description text.

4. The method of claim 2, wherein, The cross-attention fusion of the features of the noise image processed by the denoising subnetwork in the diffusion network, the features of the first image processed by the corresponding denoising subnetwork in the mirror network and the features of the description text comprises: cross-attention fusion of the features of the noise image processed by at least part of the network layers in the denoising subnetwork, the features of the first image processed by the corresponding network layers in the corresponding denoising subnetwork and the features of the description text.

5. The method of claim 1, wherein, In the case that the target object comprises a target human object, the method further comprises: obtaining a third image of a preset three-dimensional model under the preset view angle; wherein the preset three-dimensional model comprises a preset human model in a preset posture; correspondingly, the cross-attention fusion of the features of the first image, the features of the description text and the features of the noise image by the diffusion network to generate a second image comprises: generate, by a diffusion network, a second image based on features of the first image, features of the description text, features of the noisy image, and features of the third image; wherein the second image comprises an image of the target human object in the preset pose under the preset view angle.

6. The method of claim 5, wherein, After the determining the three-dimensional model of the target object, the method further comprises: binding the three-dimensional model of the target human object with a skeletal model in the preset pose; wherein the binding of the three-dimensional model of the skeletal model is driven based on pose control parameters.

7. The method of claim 1, wherein, The obtaining of the first image of the target object comprises: detecting the object in the target video to determine a video frame containing the target object; determining a target video frame from the video frame containing the target object; segmenting the target video frame to obtain the first image of the target object.

8. The method of claim 7, wherein, The obtaining of the description text corresponding to the first image comprises: generating, by an image description network, the description text corresponding to the first image based on the first image of the target object.

9. The method of claim 7, wherein, In the case that the target object comprises a target human object, the method further comprises: performing motion estimation on the target human object in the target video to determine a sequence of pose control parameters; driving the three-dimensional model of the target human object according to the sequence of pose control parameters.

10. The method of claim 7, wherein, After the determining the three-dimensional model of the target object, the method further comprises: generating a three-dimensional scene corresponding to the target video according to the three-dimensional model of the target object.

11. An apparatus for generating three-dimensional data, characterized by comprising: comprises: a first obtaining module configured to obtain a first image of a target object and a description text corresponding to the first image; a second obtaining module configured to obtain a noisy image; wherein the noisy image comprises a rendered image with added noise; the rendered image comprises an image rendered by a neural radiance field under a preset view angle; a diffusion module configured to cross-attentionally fuse features of the first image, features of the description text, and features of the noisy image by a diffusion network to generate a second image; wherein the second image comprises an image of the target object under the preset view angle; a three-dimensional representation module configured to determine a score distillation loss according to the second image, the noisy image, and the added noise, and adjust the neural radiance field according to the score distillation loss; a model generation module configured to determine a three-dimensional model of the target object according to the adjusted neural radiance field.

12. An electronic device, comprising: The electronic device comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating three-dimensional data according to any one of claims 1-10.

13. A storage medium containing computer-executable instructions for performing the method for generating three-dimensional data according to any one of claims 1-10 when executed by a computer processor.

Images