Image generation model construction method and three-dimensional digital model determination method

Abstract

The present application discloses an image generation model construction method and a three-dimensional digital model determination method. The image generation model construction method comprises: first obtaining a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image; and then using the first object description image, the first description text, and the first attitude image to construct an image generation model, so that the constructed image generation model can generate high-quality image data on the basis of the first description text and the first posture, such that a three-dimensional digital model determined by means of the image generation model can also conform to natural posture constraints.

Description

A method for constructing an image generation model and a method for determining a three-dimensional digital model

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese Patent Application No. 202310969311.8, filed on August 2, 2023. The entire teachings of the above application are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of data processing technology, and in particular to an image generation model construction method, a three-dimensional digital model determination method, an image generation method, a three-dimensional digital model display method, a device, an electronic device, and a computer-readable medium. Background Art

[0004] In some application scenarios (e.g., image generation scenarios, image style conversion scenarios, etc.), it may be necessary to pre-build a model so that the model can generate image data that meets the requirements of the application scenario (e.g., image data with a certain style, etc.). This makes how to build the model a technical problem that needs to be solved urgently.

[0005] Summary of the Invention

[0006] The present application provides an image generation model construction method, a three-dimensional digital model determination method, an image generation method, a three-dimensional digital model display method, a device, an electronic device, and a computer-readable medium.

[0007] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0008] This application provides a method for constructing an image generation model, the method comprising:

[0009] Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image;

[0010] An image generation model is constructed using the first object description image, the first description text and the first posture image. The image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0011] In one possible implementation, the process of determining the first posture image includes: determining first posture description information from the first object description image; and rendering a three-dimensional parametric model using the first posture description information to obtain a first posture image, where the three-dimensional parametric model is a three-dimensional digital model.

[0012] In a possible implementation, the first object description image is used to describe at least one part of the first object; the first posture image carries part labeling information corresponding to each part, and the part labeling information corresponding to different parts is different.

[0013] In a possible implementation manner, the colors of regions where different parts of the first posture image are located are different.

[0014] In one possible implementation, the process of constructing the image generation model includes: inputting the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model; updating the preset diffusion model based on the model output data, and continuing to execute the steps of obtaining the first object description image, the first description text corresponding to the first object description image, and the first posture image corresponding to the first object description image, until the first stop condition is reached, and determining the image generation model based on the preset diffusion model.

[0015] The present application provides a method for determining a three-dimensional digital model, the method comprising:

[0016] Rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0017] Inputting the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-built image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data based on the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image;

[0018] updating the three-dimensional representation model according to the model output data;

[0019] A three-dimensional digital model is determined according to the three-dimensional representation model.

[0020] In one possible implementation, the image generation model is constructed using the image generation model construction method provided in this application.

[0021] In one possible implementation, the image generation model is a diffusion model, and the model output data includes predicted noise;

[0022] Updating the three-dimensional representation model according to the model output data includes: determining the SDS loss corresponding to the three-dimensional representation model based on the predicted noise, the noise label corresponding to the second object description image, and the score distillation sampling SDS loss function; the noise label refers to the noise data actually used in the process of determining the model output data; and updating the three-dimensional representation model based on the SDS loss.

[0023] In a possible implementation manner, updating the three-dimensional representation model according to the model output data includes: updating parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data.

[0024] In one possible implementation manner, after updating the three-dimensional representation model according to the model output data, the method further includes:

[0025] Continuing to execute the image acquisition step, the image acquisition step at least includes the step of rendering the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0026] Determining a three-dimensional digital model based on the three-dimensional representation model includes:

[0027] When a second stopping condition is reached, a three-dimensional digital model is determined based on the three-dimensional representation model.

[0028] In a possible implementation manner, before continuing to perform the image acquisition step, the method further includes:

[0029] If the preset resolution update condition is met, the resolution of the three-dimensional representation model is updated so that the resolution of the updated three-dimensional representation model is higher than the resolution of the three-dimensional representation model before the update.

[0030] In a possible implementation manner, before continuing to perform the image acquisition step, the method further includes:

[0031] If the preset data unit filtering condition is met, a target data unit is searched for and deleted from the three-dimensional representation model, and a density parameter corresponding to the target data unit is lower than a preset density threshold.

[0032] In a possible implementation manner, the second posture description information includes an image acquisition distance;

[0033] Before continuing to perform the image acquisition step, the method further includes:

[0034] If the preset distance update condition is met, updating the image acquisition distance in the second posture description information so that the updated image acquisition distance is smaller than the image acquisition distance before the update;

[0035] The step of continuing to perform the image acquisition step includes:

[0036] Continue to perform the steps of rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

[0037] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0038] Before continuing to perform the image acquisition step, the method further includes:

[0039] If a preset focus update condition is met, updating the image acquisition focus range so that the updated image acquisition focus range is different from the image acquisition focus range before the update;

[0040] The step of continuing to perform the image acquisition step includes:

[0041] Continue to perform the steps of determining a second posture image from the three-dimensional parameterized model rendered using the second posture description information, and determining a second object description image from the three-dimensional representation model rendered using the second posture description information.

[0042] In one possible implementation, the second posture description information refers to information required to be used as a basis for acquiring an image of the second object; the image acquisition focus range before the update is used to describe the entire second object, and the image acquisition focus range after the update is used to describe the first part of the second object; or, the image acquisition focus range before the update is used to describe the second part of the second object, and the image acquisition focus range after the update is used to describe the third part of the second object.

[0043] In a possible implementation manner, before continuing to perform the image acquisition step, the method further includes:

[0044] If the preset model conversion condition is met, the three-dimensional representation model is converted from the first model representation mode to the second model representation mode.

[0045] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0046] If the image acquisition focus range is used to describe the portion to be focused, the process of determining the image acquisition focus range includes: determining the position information of the portion to be focused from the three-dimensional parametric model; and determining the image acquisition focus range based on the position information, so that the image data obtained by rendering the three-dimensional parametric model using the image acquisition focus range is used to describe the portion to be focused.

[0047] The present application provides an image generation method, the method comprising:

[0048] Acquire a third description text and third posture description information corresponding to the third description text;

[0049] Rendering the three-dimensional parameterized model using the third posture description information to obtain a third posture image;

[0050] A generated image is obtained using a pre-constructed image generation model, the third description text, and the third posture image; the image generation model is constructed using the image generation model construction method provided in this application.

[0051] The present application provides a three-dimensional digital model display method, the method comprising:

[0052] Obtaining fourth posture description information;

[0053] Rendering a predetermined three-dimensional digital model using the fourth posture description information, wherein the three-dimensional digital model is determined using the three-dimensional digital model determination method provided in this application;

[0054] A rendering result is displayed, where the rendering result satisfies the constraints described by the fourth posture description information.

[0055] This application provides an image generation model construction device, comprising:

[0056] a first acquiring unit, configured to acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image;

[0057] A model construction unit is used to construct an image generation model using the first object description image, the first description text and the first posture image, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0058] The present application provides a three-dimensional digital model determination device, comprising:

[0059] a first rendering unit, configured to render the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and to render the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0060] a data determination unit, configured to input the second object description image, the second posture image, and a second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is configured to generate image data based on the second posture image and the second description text, the image data satisfying the constraints described by the second description text and the constraints represented by the second posture image;

[0061] an information updating unit, configured to update the three-dimensional representation model according to the model output data;

[0062] The model determining unit is used to determine a three-dimensional digital model according to the three-dimensional representation model.

[0063] The present application provides an image generation device, comprising:

[0064] A second acquiring unit, configured to acquire a third description text and third posture description information corresponding to the third description text;

[0065] A second rendering unit, configured to render the three-dimensional parameterized model using the third posture description information to obtain a third posture image;

[0066] An image generation unit is used to obtain a generated image using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using the image generation model construction method provided in this application.

[0067] This application provides a three-dimensional digital model display device, comprising:

[0068] A third acquiring unit, configured to acquire fourth posture description information;

[0069] a model adjustment unit, configured to render a predetermined three-dimensional digital model using the fourth posture description information, the three-dimensional digital model being determined using the three-dimensional digital model determination method provided in this application;

[0070] A model display unit is used to display a rendering result, where the rendering result satisfies the constraints described by the fourth posture description information.

[0071] The present application provides an electronic device, the device comprising: a processor and a memory;

[0072] The memory is used to store instructions or computer programs;

[0073] The processor is used to execute the instructions or computer program in the memory so that the electronic device executes the image generation model construction method provided in this application, or executes the three-dimensional digital model determination method provided in this application, or executes the image generation method provided in this application, or executes the three-dimensional digital model display method provided in this application.

[0074] The present application provides a computer-readable medium, which stores instructions or computer programs. When the instructions or computer programs are executed on a device, the device executes the image generation model construction method provided in the present application, or executes the three-dimensional digital model determination method provided in the present application, or executes the image generation method provided in the present application, or executes the three-dimensional digital model display method provided in the present application.

[0075] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the image generation model construction method provided by the present application, or program code for executing the three-dimensional digital model determination method provided by the present application, or program code for executing the image generation method provided by the present application, or program code for executing the three-dimensional digital model display method provided by the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] FIG1 is a flowchart of a method for constructing an image generation model provided in an embodiment of the present application;

[0078] FIG2 is a schematic diagram of generated image data provided by an embodiment of the present application;

[0079] FIG3 is a flow chart of an image generation method provided in an embodiment of the present application;

[0080] FIG4 is a flow chart of a method for determining a three-dimensional digital model provided in an embodiment of the present application;

[0081] FIG5 is a schematic diagram of a process for determining a three-dimensional digital model provided in an embodiment of the present application;

[0082] FIG6 is a schematic diagram of a progressive high-resolution model generation process provided in an embodiment of the present application;

[0083] FIG7 is a flow chart of a three-dimensional digital model display method provided in an embodiment of the present application;

[0084] FIG8 is a schematic structural diagram of an image generation model construction device provided in an embodiment of the present application;

[0085] FIG9 is a schematic structural diagram of an image generating device provided in an embodiment of the present application;

[0086] FIG10 is a schematic structural diagram of a three-dimensional digital model determination device provided in an embodiment of the present application;

[0087] FIG11 is a schematic structural diagram of a three-dimensional digital model display device provided in an embodiment of the present application;

[0088] FIG12 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0090] To better understand the technical solutions provided by this application, the image generation model construction method provided by this application is described below with reference to some accompanying drawings. As shown in Figure 1, the image generation model construction method provided by the embodiment of this application includes the following steps S101-S102. Figure 1 is a flow chart of a method for constructing an image generation model provided by the embodiment of this application.

[0091] S101: Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image.

[0092] Among them, the first object description image refers to the image data required to use when constructing the image generation model below; and the first object description image is used to describe at least one part of the first object. The first object refers to the object described by the first object description image, and this application does not limit the first object. For example, the first object can be a person, an animal, or an object. It should be noted that this application does not limit the at least one part. For example, when the first object refers to a person or an animal, the at least one part of the first object can include at least one body part of the first object (for example, all body parts, or only the head, or only the legs, etc.). For another example, when the first object refers to an object (for example, a car, a building, etc.), the at least one part of the first object can include at least one component of the first object.

[0093] Furthermore, this application does not limit the method for obtaining the first object description image. For example, the first object description image may be obtained by using image data from a pre-defined training image set. It should be noted that this application does not limit the implementation of the training image set. For example, the DeepFashion (Liu et al., 2016) dataset may be used for implementation.

[0094] The first description text refers to the text information required for model training using the first object description image, so that the first description text can describe the image data generated by the model. For example, the first description text can be implemented as an image description text such as "There is a stone bench by the river."

[0095] In addition, the present application does not limit the method for obtaining the above-mentioned first description text. For example, the first description text can be pre-configured by relevant personnel for the above-mentioned first object description image, so that the first description text can be directly used when the first object description image is used to train the model later.

[0096] The first posture image is used to describe the state of the object in the first object description image above (for example, which body parts it has, what postures each body part is in, and what accessories are worn on one or more body parts, etc.); and the first posture image is determined based on the first object description image above, so that the first posture image can better represent the state of the object in the first object description image.

[0097] In addition, the present application does not limit the process of determining the first posture image above. For example, it can be implemented using any existing or future method that can determine image data that can represent the state of an object from an image data (for example, a contour extraction method, a heat map, etc.).

[0098] In fact, in order to better improve the representation effect of the object state (for example, the degree of refinement), the present application also provides a possible implementation method of the above-mentioned first posture image determination process, which may specifically include the following steps 11 and 12.

[0099] Step 11: Determine first posture description information from the first object description image.

[0100] The first posture description information is used to describe the state of the object in the first object description image (eg, which body parts the object has, what posture each body part is in, and what accessories one or more body parts are wearing, etc.).

[0101] In addition, the present application does not limit the implementation manner of the above-mentioned first posture description information. For example, it may include at least one of image acquisition information and object state representation information.

[0102] Image acquisition information refers to some parameter information (such as camera focus, camera distance and other parameter information) used when using an image acquisition device to acquire the image of the first object described above; and this application does not limit the image acquisition information. For example, in some application scenarios, the image acquisition information may at least include image acquisition distance and / or image acquisition focus range. Among them, the image acquisition distance refers to the distance between the image acquisition device and the object when the image acquisition device is used to acquire an image of an object; and this application does not limit the implementation method of the image acquisition distance. For example, in some application scenarios, when a certain model (such as the three-dimensional parameterized model or three-dimensional representation model below) is used to represent an object in a three-dimensional space, the image acquisition distance may refer to the distance between the image acquisition device and the model when the image acquisition device is used to acquire an image of the model. For example, in some application scenarios, when the image acquisition device is a camera, the image acquisition distance may be the camera distance. The image acquisition focus range is used to indicate which parts of an object the image acquisition device focuses on when the image acquisition device is used to acquire an image of the object; and the present application does not limit the implementation method of the image acquisition focus range. For example, when the image acquisition device is a camera, the image acquisition focus range can be the focus of the camera, so that subsequent adjustment processing of the camera focus can ensure that the camera can acquire images of the area that needs to be focused (for example, the entire object or a part of the object), so that the acquired image can describe the state of the object presented in the area as finely as possible (for example, the degree of finger bending, the accessories worn on the fingers, etc.) while ensuring the integrity of the area.

[0103] It should be noted that this application does not limit the implementation of the above-mentioned image acquisition device. For example, in some application scenarios, the image acquisition device can be implemented using any existing or future device capable of image acquisition (e.g., image collection, etc.) (e.g., a camera). In other application scenarios, the image acquisition device can be implemented using a virtual camera (e.g., a virtual camera in a renderer).

[0104] The object state representation information is used to describe the state of the first object presented in the first object description image; and the present application does not limit the object state representation information. For example, when the first object description image is used to describe at least one part of the first object (for example, the head, legs, etc.), the object state representation information can at least include state description information of each part. Among them, the state description information of the nth part is used to describe the state of the nth part of the first object presented in the first object description image (for example, the degree of bending, the state of the accessories worn, etc.); and the present application does not limit the implementation method of the state description information of the nth part. n is a positive integer, n≤N, N is a positive integer, and N represents the number of parts in the "at least one part".

[0105] In addition, this application does not limit the implementation of step 11 above. For example, in some application scenarios, step 11 can specifically be: after obtaining the first object description image, the posture description information corresponding to the first object description image can be searched from a pre-established mapping relationship as the first posture description information. The mapping relationship is used to record the posture description information corresponding to different image data; and this application does not limit the method for obtaining the posture description information corresponding to the image data. For example, it can be automatically generated or manually annotated by relevant personnel.

[0106] For example, in other application scenarios, step 11 above may specifically be: after acquiring the first object description image, directly using a preset posture analysis method to analyze and process the first object description image to obtain the first posture description information. The preset posture analysis method is used to analyze relevant posture information (e.g., image acquisition information and / or object state representation information) from an image data. Furthermore, this application does not limit the implementation method of the preset posture analysis method; for example, it may be implemented using any existing or future method that can analyze relevant posture information from an image data.

[0107] Based on the relevant content of step 11 above, it can be known that after obtaining the first object description image, the first posture description information can be determined from the first object description image, so that the first posture description information can describe the object state in the first object description image, so that the first posture image corresponding to the first object description image can be rendered based on the first posture description information.

[0108] Step 12: Rendering the three-dimensional parameterized model using the first posture description information to obtain a first posture image. The three-dimensional parameterized model is a three-dimensional digital model.

[0109] The three-dimensional parametric model is used to represent an object in three-dimensional space; and the three-dimensional parametric model belongs to a three-dimensional digital model.

[0110] In addition, the present application is not limited to the above three-dimensional parametric model. For example, when the three-dimensional parametric model is used to describe a person, the three-dimensional parametric model may be a human body model.

[0111] Furthermore, this application does not limit the implementation of the above-mentioned three-dimensional parametric model. For example, to further enhance the three-dimensional representation of an object, the three-dimensional parametric model may be implemented using a parametric three-dimensional model such as SMPL (Skinned Multi-Person Linear). The SMPL model includes not only the skeleton points of an object but also the object's skin, enabling the SMPL model to better represent the object's state in three-dimensional space.

[0112] Furthermore, the present application does not limit the implementation of the above step 12. For example, in order to better improve the posture representation effect, the step 12 can be implemented with the help of dense pose (DensePose; Neverova and Kokkinos 2018) technology is used to implement it. Among them, DensePose technology is a pioneering technology, and the DensePose technology helps to establish a dense correspondence between a two-dimensional image and a surface-based three-dimensional model (for example, the three-dimensional parameterized model mentioned above). For example, when the three-dimensional model is a SMPL model, the DensePose technology can assign each triangular face in the SMPL model to one of the 24 predefined body parts, so that the body part corresponding to each triangular face in the SMPL model can be determined, so that based on this correspondence, a two-dimensional body image with a part label under a certain camera focus (for example, the first posture image mentioned above) can be determined from some relevant areas presented in the SMPL model. It should be noted that the present application does not limit the implementation method of the part label. For example, it can be implemented by setting the color of the corresponding part area in the two-dimensional body image so that different part areas in the two-dimensional body image have different colors.

[0113] Based on the relevant content of step 12 above, it can be known that after obtaining the first posture description information, the three-dimensional parametric model (for example, the SMPL model) can be rendered using the first posture description information to obtain a first posture image, so that the first posture image can better represent the object state described by the first posture description information.

[0114] Based on the relevant content of steps 11 to 12 above, it can be known that after obtaining the first object description image, the first posture image corresponding to the first object description image (for example, a two-dimensional body image carrying a DensePose part label) can be determined based on the first object description image, so that the first posture image can not only indicate which parts appear in the first object description image, but also indicate the state of each part (for example, what degree of bending is reached, what kind of accessories are worn, etc.), so that the first posture image can better represent the state of the object in the first object description image.

[0115] Based on the above two paragraphs, it can be seen that in some application scenarios, when the above-mentioned first object description image is used to describe at least one part of the first object, the first posture image corresponding to the first object description image can carry at least partial labeling information corresponding to each part, and the partial labeling information corresponding to different parts is different. Among them, the partial labeling information corresponding to the nth part refers to the partial label marked for the nth part in the first posture image, so that the partial labeling information corresponding to the nth part is used to identify the nth part; and the present application does not limit the implementation method of the partial labeling information corresponding to the nth part. For example, it can be implemented in the form of text annotation, or it can be implemented by adjusting the color of the area occupied by the nth part in the first posture image. n is a positive integer, n≤N, N is a positive integer, and N represents the number of parts in the "at least one part".

[0116] Based on the content of the previous paragraph, it can be seen that in one possible implementation, if the above-mentioned first object description image is used to describe at least one part of the first object, then for the first posture image corresponding to the first object description image, the colors of the areas where different parts are located in the first posture image are different. In this way, the purpose of adding marking information to different parts of the first posture image can be achieved, so that the first posture image can not only indicate which parts appear in the first object description image, but also indicate the status of each part, thereby enabling the first posture image to better indicate the state of the object in the first object description image.

[0117] Based on the relevant content of S101 above, it can be seen that under one possible implementation method, in order to be able to train a model with better image generation performance (for example, diffusion models such as ControlNet), it is necessary not only to obtain some image data (for example, image data in the DeepFashion dataset) and the image description text corresponding to these image data, but also to obtain the posture images corresponding to these image data (for example, two-dimensional body images with DensePose part labels, etc.), so that the subsequent triplet of (image data, image description text corresponding to the image data, posture image corresponding to the image data) can be used as training data to complete the training process for the model, so that the trained model (for example, the image generation model below) can complete the image generation processing based on the image description text + posture image, so that the trained model has better image generation performance.

[0118] S102: Utilizing the first object description image, the first description text, and the first posture image, an image generation model is constructed, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0119] Among them, the image generation model refers to a model with image generation performance that is constructed using the triple (first object description image, first description text corresponding to the first object description image, first posture image corresponding to the first object description image) as training data.

[0120] In addition, this application does not limit the implementation of the above image generation model. For example, it can be implemented using any existing or future diffusion model (such as ControlNet) that can generate images based on text information + additional conditions. Among them, ControlNet (Zhang and Agrawala 2023) is a diffusion model that can generate images based on text information + additional conditions.

[0121] In addition, for the above image generation model, the image generation model can be used to generate a new image data based on a binary pair (an image description text, a posture image corresponding to the image description text), so that the image data not only conforms to the content described by the image description text, but also conforms to the object state described by the posture image, so that the image generation model can be used to complete the task of image generation processing based on text information + additional conditions.

[0122] It should be noted that the present application does not limit how to use the above image generation model to complete the image generation task. For example, in one possible implementation, the image generation model may include an image generation module, and the image generation module may be used to generate a new image data based on a binary pair (an image description text, a posture image corresponding to the image description text), so that the image data not only conforms to the content described by the image description text, but also conforms to the object state described by the posture image, so that the image generation module can subsequently complete the task of performing image generation processing based on text information + additional conditions with the help of the image generation module. It can be seen that after the binary pair (the above first description text, the above first posture image) is input into the image generation module, the image generation module can generate new image data under the constraints of the first description text and the first posture image, so that the image data satisfies both the constraints described by the first description text and the constraints represented by the first posture image.

[0123] It should also be noted that the present application does not limit the implementation method of the above-mentioned image generation module. For example, in some application scenarios, the image generation module may refer to a module existing in the image generation model that can generate a new image data based on a binary group such as (an image description text, and a posture image corresponding to the image description text). For example, when the above-mentioned image generation model is a diffusion model, the image generation module may include at least one module existing in the image generation model, and the "at least one module" may be used to implement the function of generating a new image data based on a binary group such as (an image description text, and a posture image corresponding to the image description text). It should be noted that the present application does not limit the "at least one module". For example, in some application scenarios, the "at least one module" may include a denoising module.

[0124] It should be noted that the present application does not limit the method for obtaining the image generation module in the above paragraph. For example, it can be implemented by any existing or future method that can determine the image generation module from a trained diffusion model.

[0125] It should also be noted that the present application does not limit the specific implementation method of the above “generating a new image data based on the binary group (an image description text, and the posture image corresponding to the image description text)”. For example, in some application scenarios, a new image can be generated based only on the image description text and the posture image corresponding to the image description text. For another example, in other application scenarios, a new image can be generated based on a piece of noise data, an image description text, and the posture image corresponding to the image description text. For another example, in yet other application scenarios, a new image can be generated based on an image data, an image description text, and the posture image corresponding to the image description text.

[0126] Furthermore, this application does not limit the construction process of the above-mentioned image generation model. For example, when the image generation model is a diffusion model (e.g., ControlNet), the construction process of the image generation model can be implemented using any existing or future method that can train a diffusion model based on training data. For ease of understanding, the following example is used to illustrate.

[0127] As an example, when the above image generation model is a diffusion model (such as ControlNet), the construction process of the image generation model may specifically include the following steps 21 to 23.

[0128] Step 21: Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image.

[0129] It should be noted that the relevant content of step 21 can be found in S101 above, and for the sake of brevity, it will not be repeated here.

[0130] Step 22: Input the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model.

[0131] The preset diffusion model refers to a diffusion model required for use when constructing an image generation model; the preset diffusion model is used to perform image generation processing based on the input data of the preset diffusion model (e.g., the triples shown above). Furthermore, this application does not limit the implementation of the preset diffusion model; for example, it can be implemented using ControlNet.

[0132] Model output data refers to the data (e.g., generated image data and / or predicted noise) obtained by the preset diffusion model after inputting the triplet (first object description image, first description text corresponding to the first object description image, and first posture image corresponding to the first object description image) into the preset diffusion model. The predicted noise refers to the noise data predicted by the preset diffusion model and used in the noise addition stage of the preset diffusion model.

[0133] Step 23: Update the preset diffusion model according to the above model output data, and continue to execute the above step 21 and subsequent steps until the first stop condition is reached, and then determine the image generation model according to the preset diffusion model.

[0134] The first stopping condition refers to a condition that must be met to terminate the training process for the preset diffusion model. The first stopping condition can be pre-set. For example, the first stopping condition may include: the model loss of the preset diffusion model is lower than a pre-set first loss threshold. Another example may include: the rate of change of the model loss of the preset diffusion model is lower than a pre-set first rate of change threshold. Another example may include: the number of updates to the preset diffusion model reaches a pre-set first number threshold.

[0135] The model loss of the preset diffusion model is used to characterize the image generation performance of the preset diffusion model; and the model loss of the preset diffusion model is typically determined based on the model output data of the preset diffusion model (e.g., generated image data and / or predicted noise). Furthermore, this application does not limit the method for calculating the model loss of the preset diffusion model; it may be implemented using any existing or future loss function required for training a diffusion model.

[0136] It should be noted that this application does not limit the implementation method of the step "updating the preset diffusion model based on the model output data" in step 23 above. For example, it can be implemented using any existing or future diffusion model updating method. As an example, the step "updating the preset diffusion model based on the model output data" can specifically include: first calculating the model loss of the preset diffusion model based on the model output data of the preset diffusion model; then updating the model parameters of the preset diffusion model based on the model loss, so that the updated parameter preset diffusion model has better image generation performance.

[0137] Based on the relevant content of steps 21 to 23 above, it can be seen that in some application scenarios, the present application can use an image dataset (e.g., DeepFashion dataset), text information corresponding to each image data in the image dataset, and posture images corresponding to each image data in the image dataset (e.g., two-dimensional body images with DensePose part labels, etc.) to train ControlNet, so that the trained ControlNet has better image generation performance. For example, the image data generated by the trained ControlNet shown in Figure 2 can fully demonstrate the ControlNet's ability to generate high-quality images with consistent perspectives (e.g., including various camera focuses and various body parts, such as the whole body, legs, head, etc.), so that when a certain posture description information is given (e.g., a certain camera focus, one or more body parts in a certain posture, etc.), the SMPL model with part labels can be rendered with the help of the posture description information to generate a posture image (e.g., a two-dimensional body image with DensePose part labels, etc.); and then the posture image is input into the trained ControlNet as a condition, so that the trained ControlNet can generate image data that meets the condition (e.g., the image data shown in Figure 2).

[0138] Based on the relevant contents of S101 to S102 above, it can be seen that for the image generation model construction method provided in the embodiment of the present application, the first object description image (for example, image data in a certain training image set), the first description text corresponding to the first object description image (for example, image description text similar to "There is a stone bench by the river"), and the first posture image corresponding to the first object description image (for example, a dense posture rendering image, etc.) are first obtained; then the first object description image, the first description text and the first posture image are used to construct an image generation model, so that the constructed image generation model can generate relatively high-quality image data based on the first description text and the first posture image, so that the image generation model can achieve the purpose of generating image data based on text information and posture information, which is conducive to improving the image generation effect of the image generation model. Among them, because the above-mentioned first posture image is determined based on the first object description image, the first posture image can better represent the state of the object in the first object description image (for example, which parts exist and what postures each part is in, etc.), so that the image data generated with the first posture image as a guiding condition is more in line with the natural posture constraints corresponding to different parts of the object (for example, the head, etc.), and thus the image data generated with the first posture image as a guiding condition has higher quality, which can effectively avoid the problems caused when generating image data based only on text information, thereby helping to improve the image generation effect.

[0139] In addition, the present application does not limit the execution subject of the above image generation model construction method. For example, any embodiment of the image generation model construction method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For example, any embodiment of the image generation model construction method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer, etc. The server can be a stand-alone server, a cluster server or a cloud server.

[0140] In some application scenarios, after constructing the above image generation model, the image generation model can be used to implement image processing tasks. Based on this, the present application also provides an image generation method, as shown in Figure 3, which may include the following S301-S303. Figure 3 is a flowchart of an image generation method provided in an embodiment of the present application.

[0141] S301: Acquire a third description text and third posture description information corresponding to the third description text.

[0142] The third description text refers to the text required to generate a new image data using the constructed image generation model. It should be noted that the implementation of the third description text is similar to the implementation of the first description text above, and for the sake of brevity, it will not be repeated here.

[0143] The third posture description information refers to the posture description information required when generating new image data using the constructed image generation model, and corresponds to the third description text above. It should be noted that the implementation of this third posture description information is similar to the implementation of the first posture description information above, and for the sake of brevity, it will not be repeated here.

[0144] S302: Rendering the three-dimensional parameterized model using the third posture description information to obtain a third posture image.

[0145] The third posture image is used to display the object state described by the third posture description information. It should be noted that the implementation of the third posture image is similar to the implementation of the first posture image above, and for the sake of brevity, it will not be repeated here.

[0146] Based on the relevant content of S302 above, it can be known that for some application scenarios, after obtaining the third posture description information, the third posture description information can be used to render the above three-dimensional parametric model (for example, the SMPL model) to obtain a third posture image, so that the third posture image can better represent the object state described by the third posture description information.

[0147] S303: Obtain a generated image using a pre-constructed image generation model, a third description text, and a third posture image; the image generation model is constructed using any implementation of the image generation model construction method provided in this application.

[0148] In the present application, after obtaining the above-mentioned third description text and its corresponding third posture image, the binary group (the third description text, the third posture image) can be input into a pre-constructed image generation model (for example, the image generation module in the image generation model), so that the image generation model (for example, the image generation module in the image generation model) can perform image generation processing based on the third description text and the third posture image, obtain and output the generated image, so that the generated image conforms to both the constraints described in the third description text (for example, constraints on image content) and the constraints described in the third posture image (for example, constraints on object state, for example, natural posture constraints of some body parts and / or jewelry wearing constraints, etc.). In this way, defects caused by the generated image data not conforming to the natural posture constraints of certain body parts (for example, generating inferior image data that does not conform to natural laws) can be effectively avoided, which is conducive to improving the image generation effect.

[0149] Based on the relevant contents of S301 to S303 above, it can be known that for some application scenarios, after obtaining the constructed image generation model, the image generation model can be used to complete the task of performing image generation processing based on text + additional conditions, so that the generated image data also meets the constraints described by the text + additional conditions. Among them, because the image generation model performs image generation processing based on text + posture image, the image data generated by the image generation model conforms to both the image description content defined by the text and the object state defined by the posture image (for example, the natural posture constraints of some body parts, etc.). This can effectively avoid defects caused by the generated image data not conforming to the natural posture constraints of some body parts (for example, generating low-quality image data that does not conform to natural laws), which is conducive to improving the image generation effect.

[0150] In addition, this application does not limit the execution subject of the above image generation method. For example, any implementation of the image generation method provided in the embodiments of this application can be applied to a device with data processing capabilities, such as a terminal device or a server. For another example, any implementation of the image generation method provided in the embodiments of this application can also be implemented through a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0151] In fact, for the image generation model constructed above, this image generation model can not only be used to implement image generation tasks, but also to implement three-dimensional digital model determination tasks (for example, three-dimensional digital human determination tasks). Based on this, the present application also provides a three-dimensional digital model determination method, as shown in Figure 4. The three-dimensional digital model determination method may include the following steps S401-S404. Figure 4 is a flowchart of a three-dimensional digital model determination method provided in an embodiment of the present application.

[0152] S401: Rendering a three-dimensional parameterized model using second posture description information to obtain a second posture image, and rendering a three-dimensional representation model using the second posture description information to obtain a second object description image.

[0153] The second pose description information refers to the pose description information involved in determining the three-dimensional digital model, so that the second pose description information can be used as pose description information shared by the three-dimensional parametric model and the three-dimensional representation model. It should be noted that the implementation of the second pose description information is similar to the implementation of the first pose description information described above, and for the sake of brevity, it will not be repeated here.

[0154] In addition, the present application does not limit the method for obtaining the second posture description information. For example, it can be set by relevant personnel according to actual application scenarios.

[0155] A three-dimensional parametric model refers to a three-dimensional digital model with fixed parameters involved in the determination process of a three-dimensional digital model (for example, the three-dimensional parametric model shown in FIG5 ), so that the three-dimensional digital model can provide prior information (for example, the dense pose rendering image shown in FIG5 ) during the determination process of the three-dimensional digital model; and the relevant content of the three-dimensional parametric model can be found above.

[0156] The second pose image is used to display the object state described by the second pose description information. Furthermore, the second pose image refers to image data obtained by rendering the three-dimensional parameterized model using the second pose description information (e.g., the dense pose rendering image shown in FIG5 ). It should be noted that the implementation of the second pose image is similar to that of the first pose image described above and, for the sake of brevity, is not further described here.

[0157] It can be seen that, under a possible implementation method, the acquisition process of the second posture image above can be specifically as follows: after obtaining the second posture description information, the three-dimensional parametric model can be rendered using the second posture description information to obtain the second posture image, so that the second posture image can better represent the object state defined by the second posture description information.

[0158] A three-dimensional representation model refers to a three-dimensional digital model involved in the determination of a three-dimensional digital model and requiring data unit parameter update processing (e.g., the three-dimensional representation model shown in FIG5 ); and this application does not limit the implementation method of the three-dimensional representation model. For example, it can be implemented using a neural radiation field (Nerf) grid. The data unit refers to a data storage unit present in the three-dimensional representation model; and this application does not limit the implementation method of the data unit. For example, when the three-dimensional representation model is implemented using a Nerf grid, the data unit refers to a spatial grid present in the Nerf grid.

[0159] The second object description image refers to image data required for use in determining the three-dimensional digital model; and the second object description image may refer to image data obtained by rendering the three-dimensional representation model using the second posture description information (e.g., the mesh rendering image shown in FIG5 ), so that the second object description image is used to describe at least a portion of the second object. The second object refers to the object described by the second object description image. It should be noted that the implementation of the second object is similar to the implementation of the first object described above, and for the sake of brevity, it will not be repeated here.

[0160] It can be seen that, in one possible implementation, the process of acquiring the second object description image above can be specifically as follows: after acquiring the second posture description information, the second posture description information can be used to render the three-dimensional representation model to obtain the second object description image, so that the second object description image can better describe at least one part of the second object.

[0161] It should be noted that this application does not limit the relationship between the execution time of the above-mentioned second posture image acquisition process and the execution time of the above-mentioned second object description image acquisition process. For example, in some application scenarios, the former may be earlier than the latter. For another example, in other application scenarios, the latter may be earlier than the former. For another example, in still other application scenarios, there is almost no difference between the two.

[0162] Based on the relevant content of S401 above, it can be known that in the process of determining the three-dimensional digital model, in order to better ensure the determination effect of the three-dimensional digital model, the three-dimensional parametric model and the three-dimensional representation model can be rendered using the same posture description information, so that the second posture image obtained by rendering based on the three-dimensional parametric model and the second object description image obtained by rendering based on the three-dimensional representation model can display the object at the same perspective, so that the second posture image and the second object description image can present a perspective alignment state, and thus achieve a coherent and consistent effect between the object state described by the second posture image and the object state presented in the second object description image. In this way, the second posture image can represent the object state in the second object description image, so that the second posture image and the second object description image can be used to construct a training data subsequently, and the training data can be used to participate in the determination process of the three-dimensional digital model. In this way, the determination process of the three-dimensional digital model can be completed without the need for additional label information.

[0163] S402: Input the second object description image, the second posture image and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data based on the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image.

[0164] The second description text refers to the textual information required for image generation processing using the second object description image described above, so that the second description text can represent the generated image data. It should be noted that the implementation of the second description text is similar to the implementation of the first description text described above, and for the sake of brevity, it will not be repeated here.

[0165] In addition, this application does not limit the method for obtaining the second description text above. For example, it can be implemented using a text provided by relevant personnel.

[0166] The image generation model is used to perform image generation processing based on an image description text and a posture image corresponding to the image description text. For example, the image generation model can perform image generation processing based on a triplet of (image data, image description text corresponding to the image data, and posture image corresponding to the image data); and the image generation model is constructed using any embodiment of the image generation model construction method provided in this application. It should be noted that the relevant content of the image generation model can be found above, and for the sake of brevity, it will not be repeated here.

[0167] The model output data refers to the result (for example, generated image data and / or predicted noise) obtained by the image generation model performing image generation processing based on the triplet (second object description image, second posture image, second description text).

[0168] Based on the relevant content of S402 above, it can be known that for the current round of processing, after obtaining the second object description image, the second posture image corresponding to the second object description image, and the second description text corresponding to the second object description image, the triplet (second object description image, second posture image, second description text) can be input into a pre-built image generation model, so that the image generation model can perform image generation processing based on the triplet, obtain and output model output data (for example, generated image data and / or predicted noise data), so that the performance of the above three-dimensional representation model can be determined based on the model output data in the future.

[0169] S403: Update the three-dimensional representation model according to the model output data.

[0170] It should be noted that the present application does not limit the implementation method of the above S403. For example, when the above image generation model belongs to a diffusion model and the model output data of the image generation model includes predicted noise, the S403 may specifically include the following steps 31-32.

[0171] Step 31: Determine the SDS loss corresponding to the three-dimensional representation model based on the predicted noise in the model output data of the image generation model, the noise label corresponding to the second object description image, and the SDS loss function. The noise label refers to the noise data actually used in the process of determining the model output data.

[0172] The predicted noise refers to the noise data predicted and used when the above image generation model performs image generation processing on the triplet (second object description image, second posture image, second description text).

[0173] The noise label refers to the noise data actually used when the above image generation model performs image generation processing on the triplet (second object description image, second posture image, second description text).

[0174] SDS refers to a method proposed by DreamFusion (Poole et al. 2022) to assist in generating a three-dimensional digital model based on a two-dimensional diffusion model; and the SDS loss function can be implemented using the SDS loss function shown in the following formulas (1)-(2).

[0175] Wherein, x = g(θ, P) represents the second object description image determined by using the three-dimensional representation model, g represents the rendering function of the three-dimensional representation model, θ represents the parameter information corresponding to each data unit in the three-dimensional representation model, P represents the second pose image mentioned above, and g(θ, P) represents the three-dimensional representation model rendered according to the second pose image; c = h(SMPL, P) represents the second pose image determined by using a three-dimensional parameterized model (for example, an SMPL model), h represents the rendering function of the three-dimensional parameterized model, SMPL represents that the three-dimensional parameterized model is implemented using the SMPL model; y represents the second description text mentioned above; t represents the noise level, and z t represents the noise result obtained by the noise addition module in the image generation model for the image data x; w(t) represents the weight information determined according to t and y; ∈ φ Indicates the image generation model (e.g., ControlNet) constructed above, which is used for the noise addition result z t The processing module (for example, denoising module, etc.), φ represents the ∈ φ Parameters present in ; Represents the predicted noise; ∈ represents the random noise data actually added when the image generation model performs noise processing on the image data x; represents the SDS loss function, is the differential symbol, is the partial derivative symbol.

[0176] The SDS loss corresponding to the above three-dimensional representation model is used to characterize the performance of the three-dimensional representation model, and the SDS loss can be calculated using the SDS loss function shown in the above formulas (1)-(2).

[0177] Step 32: Update the three-dimensional representation model according to the SDS loss corresponding to the above three-dimensional representation model.

[0178] In this application, after obtaining the SDS loss corresponding to the above-mentioned three-dimensional representation model, the three-dimensional representation model can be updated based on the SDS loss (for example, the parameter information corresponding to each data unit in the three-dimensional representation model is updated) so that the updated three-dimensional representation model can better represent the object state. It should be noted that the relevant content of this parameter information is shown below.

[0179] Based on the relevant content of steps 31 to 32 above, it can be seen that for some application scenarios, after obtaining the model output data obtained by the image generation model for the triple (second object description image, second posture image, second description text), the SDS loss corresponding to the three-dimensional representation model can be determined based on the model output data and the SDS loss function; and then the three-dimensional representation model can be updated based on the SDS loss so that the updated three-dimensional representation model has better performance.

[0180] In fact, in some application scenarios, each data unit in the above three-dimensional representation model corresponds to some parameter information, and these parameter information can affect the object described by the three-dimensional representation model (for example, the posture of some body parts of the object, the accessories worn, the clothes worn, etc.), so in order to better improve the training effect of the three-dimensional representation model, these parameter information can be updated. Based on this, the present application also provides a possible implementation of S403 above, which may specifically include: updating the parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data of the above image generation model. Among them, for any data unit in the three-dimensional representation model, the parameter information corresponding to the data unit is used to characterize the information carried by the data unit; and the present application does not limit the parameter information corresponding to the data unit, for example, the parameter information corresponding to the data unit may include the density parameter corresponding to the data unit and the color parameter corresponding to the data unit.

[0181] Based on the content in the previous paragraph, it can be seen that for some application scenarios, after obtaining the model output data obtained by the image generation model for the triple (second object description image, second posture image, second description text), the SDS loss corresponding to the three-dimensional representation model can be determined based on the model output data and the SDS loss function; and then the parameter information corresponding to each data unit in the three-dimensional representation model is updated based on the SDS loss, so that the updated parameter information can better represent the object state, thereby enabling the three-dimensional representation model with these parameter information to achieve better performance.

[0182] It should be noted that for any data unit in the three-dimensional representation model, this application does not limit the process of determining the parameter information corresponding to the data unit. For example, when the three-dimensional representation model is implemented using a Nerf grid, the process of obtaining the parameter information corresponding to the data unit can be specifically: using a preset visualization algorithm (such as explicit Nerf) to visualize the Nerf grid and obtain at least one view stereo (such as the view stereo corresponding to the density parameter and the view stereo corresponding to the color parameter shown in Figure 5); then using a preset multi-layer perception algorithm to process these view stereos to obtain the density parameters and color parameters corresponding to each grid (that is, data unit) in the Nerf grid. The preset visualization algorithm is used to visualize the Nerf grid; and this application does not limit the preset visualization algorithm. For example, it can be implemented using methods such as explicit Nerf or implicit Nerf. The preset multi-layer perception algorithm is used to extract corresponding parameter information from these view stereos respectively; and this application does not limit the implementation method of the preset multi-layer perception algorithm. For example, it can be implemented using a shallow multilayer perceptron (Shallow MLP).

[0183] Based on the relevant content of S403 above, it can be known that for the current round of processing, after obtaining the model output data obtained by the image generation model for the triplet (second object description image, second posture image, second description text), the three-dimensional representation model can be updated based on the model output data so that the updated three-dimensional representation model has better performance.

[0184] S404: Determine a three-dimensional digital model according to the updated three-dimensional representation model.

[0185] In the present application, after obtaining the updated three-dimensional representation model, the updated three-dimensional representation model can be directly determined as a three-dimensional digital model, so that the three-dimensional digital model can better represent the state of the object.

[0186] Based on the relevant contents of S401 to S404 above, it can be known that for some application scenarios, after obtaining the constructed image generation model, the image generation model can perform image generation processing for any perspective (for example, a certain camera focus + some body parts in a certain posture, etc.), so that the image data at any perspective generated based on the image generation model conforms to the natural posture constraints corresponding to the perspective, so that the three-dimensional digital model determined based on the image generation model can more accurately represent the state of the object at different perspectives, thereby making the three-dimensional digital model have better performance, which is conducive to improving the determination effect of the three-dimensional digital model.

[0187] In fact, in order to better improve the determination effect of the three-dimensional digital model, the determination process for the three-dimensional digital model can be completed by using multiple iterative updates. Based on this, the present application also provides a possible implementation method of the above three-dimensional digital model determination method, which can specifically include the following steps 41-43.

[0188] Step 41: Rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

[0189] It should be noted that for the relevant content of step 41, please refer to the relevant content of S401 above.

[0190] Step 42: Input the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-built image generation model to obtain model output data of the image generation model.

[0191] It should be noted that for the relevant content of step 42, please refer to the relevant content of S402 above.

[0192] Step 43: Update the 3D representation model based on the model output data of the image generation model, and return to the image acquisition step and subsequent steps until a second stopping condition is met, at which point a 3D digital model is determined based on the 3D representation model. The image acquisition step may include at least the step of "rendering the 3D representation model using the second pose description information to obtain a second object description image" in step 41 above.

[0193] Among them, the image acquisition step is used to realize the process of obtaining image data by rendering the above three-dimensional representation model (and three-dimensional parameterized model); and the image acquisition step at least includes the step of "using the second posture description information to render the three-dimensional representation model to obtain a second object description image" in the above step 41.

[0194] In addition, this application does not limit the implementation of the above image acquisition step. For example, in some application scenarios, the image acquisition step may refer to the above step 41. Based on this, it can be seen that in some cases, the above step 43 may specifically be: updating the three-dimensional representation model based on the model output data of the above image generation model, and returning to continue executing step 41 and subsequent steps until the second stop condition is met, and then determining the three-dimensional digital model based on the three-dimensional representation model.

[0195] In fact, the parameters in the above three-dimensional parametric model remain unchanged. Therefore, in order to better save resources (such as time, memory, computing resources, etc.), when the second posture description information does not change, the step of "determining the second posture image from the three-dimensional parametric model rendered using the second posture description information" can be omitted in the new round of processing. Based on this, the present application also provides another possible implementation of step 43 above, which can be specifically: updating the three-dimensional representation model based on the model output data of the above image generation model, and returning to continue executing the step of "rendering the three-dimensional representation model using the second posture description information to obtain the second object description image" in step 41 above and its subsequent steps until the second stop condition is reached, and then determining the three-dimensional digital model based on the three-dimensional representation model.

[0196] The second stopping condition refers to a condition that must be met at the end of the determination process of the three-dimensional digital model. The second stopping condition can be pre-set. For example, the second stopping condition may include: the SDS loss corresponding to the three-dimensional digital model is lower than a pre-set second loss threshold. In another example, the second stopping condition may include: the rate of change of the SDS loss corresponding to the three-dimensional digital model is lower than a pre-set second change rate threshold. In another example, the second stopping condition may include: the number of updates to the three-dimensional digital model reaches a pre-set second number threshold.

[0197] In addition, for the relevant content of the step "updating the three-dimensional representation model according to the model output data of the image generation model" in the above step 43, please refer to the relevant content of S403 above, and for the relevant content of the step "determining the three-dimensional digital model according to the three-dimensional representation model" in the above step 43, please refer to the relevant content of S404 above.

[0198] Furthermore, this application does not limit the timing of determining the second stop condition. For example, in some application scenarios, the step of "determining whether the second stop condition has been met" may be performed earlier than the step of "updating the three-dimensional representation model based on the model output data of the image generation model" in step 43. That is, first determine whether the second stop condition has been met. If not, update the three-dimensional representation model based on the model output data of the image generation model, and return to continue executing the image acquisition step (e.g., part or all of step 41) and its subsequent steps. If the second stop condition has been met, determine the three-dimensional digital model based on the three-dimensional representation model.

[0199] For example, in some application scenarios, the step of "determining whether the second stopping condition has been met" may be executed later than the step of "updating the three-dimensional representation model based on the model output data of the image generation model" in step 43. That is, after acquiring the model output data of the image generation model, the three-dimensional representation model is first updated based on the model output data, and then a determination is made as to whether the second stopping condition has been met. If the second stopping condition has not been met, the process returns to the image acquisition step (e.g., part or all of step 41) and its subsequent steps. If the second stopping condition has been met, the three-dimensional digital model is determined based on the three-dimensional representation model.

[0200] For example, in some application scenarios, the execution time of the step of "determining whether the second stop condition is reached" is almost close to the execution time of the step of "updating the three-dimensional representation model according to the model output data of the above image generation model" in step 43.

[0201] Based on the relevant content of steps 41 to 43 above, it can be seen that in some application scenarios, a pre-built image generation model can be used to assist in completing multiple rounds of iterative update processes for the three-dimensional representation model, so that the three-dimensional digital model can be determined based on the final three-dimensional representation model in the future, so that the three-dimensional digital model can better display the object, which is conducive to improving the determination effect of the three-dimensional digital model.

[0202] In fact, in order to better improve the determination effect of the three-dimensional digital model, it is necessary not only to update the parameter information corresponding to each data unit in the above three-dimensional representation model, but also to update some other information (for example, the resolution of the three-dimensional representation model, the above second posture description information, etc.), which is conducive to further improving the refinement of the three-dimensional digital model.

[0203] Based on the above content, it can be seen that the present application also provides a possible implementation of the above three-dimensional digital model determination method. Under this implementation, the three-dimensional digital model determination method may at least include the following steps 51 to 53.

[0204] Step 51: determining a second pose image from a three-dimensional parameterized model rendered using the second pose description information, and determining a second object description image from a three-dimensional representation model rendered using the second pose description information.

[0205] It should be noted that for the relevant content of step 51, please refer to the relevant content of S401 above.

[0206] Step 52: Input the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-built image generation model to obtain model output data of the image generation model.

[0207] It should be noted that for the relevant content of step 52, please refer to the relevant content of S402 above.

[0208] Step 53: Execute at least one information update operation and return to the image acquisition step and subsequent steps until a second stop condition is met, whereupon a three-dimensional digital model is determined based on the three-dimensional representation model. The image acquisition step may include at least the step of "rendering the three-dimensional representation model using the second pose description information to obtain a second object description image" in step 51 above.

[0209] For the relevant content of the image acquisition step, please refer to step 43 above, and for the sake of brevity, it will not be repeated here.

[0210] The information update operation is used to implement the update process of certain information (for example, parameter information of each data unit in the three-dimensional representation model, the resolution of the three-dimensional representation model, some parameters involved in the image acquisition device, the model expression of the three-dimensional representation model, etc.).

[0211] In addition, the present application does not limit the above "at least one information update operation". For example, in some application scenarios, the "at least one information update operation" may at least include a data unit parameter update operation. The data unit parameter update operation is used to implement the update process of the parameter information corresponding to each data unit in the three-dimensional representation model. It can be seen that in one possible implementation, the data unit parameter update operation is used to implement the above step of "updating the parameter information corresponding to each data unit in the three-dimensional representation model based on the model output data of the above image generation model".

[0212] In order to better determine a more refined 3D digital model, this application also provides an implementation of the aforementioned "at least one information update operation." In this implementation, the "at least one information update operation" includes not only the aforementioned data unit parameter update operation, but also one or more of a resolution enhancement operation, a data unit filtering operation, a distance reduction operation, a focus range update operation, and a model expression conversion operation. These operations are described below.

[0213] The resolution boosting operation is used to increase the resolution of the three-dimensional representation model, so that the resolution of the three-dimensional representation model can be gradually improved as the three-dimensional representation model is iteratively updated. Based on this, the present application also provides a possible implementation of the update process in step 53 above, which may include at least step 531 below.

[0214] Step 531: If the preset resolution update condition is met, the resolution of the 3D representation model is updated so that the resolution of the updated 3D representation model is higher than the resolution of the 3D representation model before the update.

[0215] The preset resolution update condition refers to the condition that needs to be met when the resolution of the three-dimensional representation model needs to be increased; and the preset resolution update condition can be pre-set. For example, in some application scenarios, the preset resolution update condition can specifically be: the SDS loss corresponding to the three-dimensional representation model reaches a third loss threshold. For another example, in some application scenarios, the preset resolution update condition can specifically be: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the third number threshold. Based on this, it can be seen that between two adjacent resolution increase processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the third number threshold.

[0216] That is, when the third numerical threshold value above is F, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Fth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the resolution of the three-dimensional representation model needs to be increased; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Fth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the resolution of the three-dimensional representation model needs to be increased; ... (and so on). Wherein, F is a positive integer, and F can be set according to the actual application scenario.

[0217] It should be noted that the present application does not limit the implementation method of increasing the resolution of the three-dimensional representation model. For example, the increase process can be implemented using the following formula (3).

[0218] The resolution of the updated 3D representation model = the resolution of the 3D representation model before the update + the preset resolution step size (3)

[0219] The preset resolution step size refers to a single increase in resolution that is set in advance based on an application scenario.

[0220] In fact, in some application scenarios, in order to avoid excessively high resolution, a resolution upper limit can be set. Based on this, the present application also provides two possible implementations of the above preset resolution update condition. In one possible implementation, the preset resolution update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches a third loss threshold, and the resolution of the three-dimensional representation model does not exceed the preset resolution threshold. In another possible implementation, the preset resolution update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the third number threshold, and the resolution of the three-dimensional representation model does not exceed the preset resolution threshold.

[0221] Based on the relevant content of step 531 above, it can be seen that in some application scenarios, for the information update involved in certain rounds of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model can be updated, but also the resolution of the three-dimensional representation model can be increased. In this way, the resolution of the three-dimensional representation model can be gradually improved with the iterative update process of the three-dimensional representation model. This gradual change process can be achieved, so that the three-dimensional representation model can represent the object state more and more clearly, which is conducive to improving the determination effect of the three-dimensional digital model.

[0222] Based on the relevant content of the resolution enhancement operation above, it can be seen that in one possible implementation method, the resolution enhancement operation is used to implement the above step of "if the preset resolution update conditions are met, then the resolution of the three-dimensional representation model is updated so that the resolution of the updated three-dimensional representation model is higher than the resolution of the three-dimensional representation model before the update", so that the resolution enhancement operation can be used to complete the gradual improvement of the model resolution in the future.

[0223] The data unit filtering operation is used to implement the data unit filtering process for the three-dimensional representation model, so that the data unit filtering operation can be used to complete the gradual change process of gradually reducing the number of data units of the three-dimensional representation model as the three-dimensional representation model is iteratively updated. This is conducive to improving the iterative update efficiency of the three-dimensional representation model. Based on this, the present application also provides a possible implementation method of the update process in step 53 above. Under this implementation method, when the parameter information corresponding to each data unit in the three-dimensional representation model includes at least density information, step 53 can at least include the following step 532.

[0224] Step 532: Search and delete a target data unit from the three-dimensional representation model, where the density parameter corresponding to the target data unit is lower than a preset density threshold.

[0225] The target data unit refers to a data unit existing in the above three-dimensional representation model and having a density parameter lower than a preset density threshold; and this application does not limit the implementation method of the target data unit.

[0226] The preset density threshold refers to the threshold required for screening the worthless data units existing in the above three-dimensional representation model; and the present application does not limit the implementation method of the preset density threshold. For example, the preset density threshold may refer to a fixed value set in advance according to the application scenario. For another example, in some application scenarios, the preset density threshold may gradually increase according to a preset rule, and the present application does not limit the preset rule. For example, it may specifically be: when the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the preset number threshold, the preset density threshold is updated so that the updated preset density threshold is greater than the preset density threshold before the update (for example, the updated preset density threshold = the preset density threshold before the update + the preset density increment).

[0227] In addition, the present application does not limit the execution method of the above step 532. For example, in one possible implementation, step 532 can be executed once in each round of processing, so that the purpose of performing a data unit filtering operation once in each round of processing can be achieved.

[0228] In fact, in order to improve efficiency, the number of executions of the data unit filtering operation can be reduced. Based on this, the present application also provides a possible implementation of the above step 532, which can be specifically: if the preset data unit filtering conditions are met, then search and delete the target data unit from the three-dimensional representation model.

[0229] Among them, the preset data unit filtering condition refers to the condition that needs to be met when performing data unit filtering processing on the three-dimensional representation model; and the preset data unit filtering condition can be pre-set. For example, in some application scenarios, the preset data unit filtering condition can specifically be: the SDS loss corresponding to the three-dimensional representation model reaches the fourth loss threshold. For another example, in some application scenarios, the preset data unit filtering condition can specifically be: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fourth number threshold. Based on this, it can be seen that between two adjacent data unit filtering processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the fourth number threshold.

[0230] That is, when the fourth threshold value above is J, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Jth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the data unit filtering process needs to be performed on the three-dimensional representation model; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Jth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the data unit filtering process needs to be performed on the three-dimensional representation model; ... (and so on). Wherein, J is a positive integer, and J can be set according to the actual application scenario.

[0231] In fact, in some application scenarios, in order to avoid deleting too many data units, a lower limit on the number of data units can be set. Based on this, the present application also provides two possible implementations of the above-mentioned preset data unit filtering conditions. In one possible implementation, the preset data unit filtering condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the fourth loss threshold, and the number of data units in the three-dimensional representation model is not less than the preset number threshold. In another possible implementation, the preset data unit filtering condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fourth number threshold, and the number of data units in the three-dimensional representation model is not less than the preset number threshold.

[0232] In addition, in order to avoid the adverse effects caused by the accidental deletion of data units, the present application also provides a possible implementation method of the above-mentioned target data unit. Under this implementation method, the target data unit meets the following conditions: the density parameter corresponding to the target data unit is lower than the preset density threshold, and the target data unit is located in the edge area of ​​the above-mentioned three-dimensional representation model. In this way, the purpose of deleting the data units that appear in the edge area of ​​the three-dimensional representation model and have a density parameter lower than the preset density threshold can be achieved with the help of this condition. In this way, the problems caused by the accidental deletion of data units in the non-edge area of ​​the three-dimensional representation model can be effectively avoided, thereby helping to improve the training effect of the three-dimensional representation model. It should be noted that the present application does not limit the method for determining the edge area. For example, it can determine its edge area from the three-dimensional representation model according to a pre-set area division rule.

[0233] Based on the relevant content of step 532 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only can the parameter information corresponding to each data unit in the three-dimensional representation model be updated, but the data unit filtering processing can also be performed on the three-dimensional representation model to delete the data units below the preset density threshold that appear in the edge area of ​​the three-dimensional representation model. In this way, the three-dimensional representation model after filtering can present the effect of the object area being tightly surrounded by the bounding box (bbox), so that the worthless data units in the three-dimensional representation model (that is, data units that are not used to represent the object state) are as few as possible, and thus the three-dimensional representation model after filtering can display the object with as few data units as possible. In this way, the number of data units of the three-dimensional representation model gradually decreases with the iterative update process of the three-dimensional representation model. This can effectively reduce the number of data units of the three-dimensional representation model, which is conducive to improving the determination speed of the three-dimensional digital model.

[0234] It should be noted that the present application does not limit the implementation method of the above data unit filtering processing. For example, it can be implemented by any method that can filter out data units whose density parameters are lower than a preset density threshold from the three-dimensional representation model (for example, the bounding box clipping method shown in Figure 6).

[0235] Based on the relevant content of the data unit filtering operation above, it can be seen that in one possible implementation method, the data unit filtering operation can be used to implement the above step of "if the preset data unit filtering conditions are met, then find and delete the target data unit from the three-dimensional representation model, and the density parameter corresponding to the target data unit is lower than the preset density threshold", so that the data unit filtering operation can be used to complete the gradual reduction of the number of model data units.

[0236] The distance shortening operation is used to shorten the distance between the image acquisition device and the three-dimensional representation model, so that the image acquisition device can subsequently gradually approach the three-dimensional representation model as the three-dimensional representation model is iteratively updated. Based on this, the present application also provides a possible implementation of the update process in step 53 above. In this implementation, when the second posture description information above includes the image acquisition distance, step 53 can at least include step 533 below.

[0237] Step 533: If the preset distance update condition is met, the image acquisition distance in the second posture description information is updated so that the updated image acquisition distance in the second posture description information is smaller than the image acquisition distance before the update.

[0238] Among them, the preset distance update condition refers to the condition that needs to be met when the distance between the image acquisition device and the three-dimensional representation model needs to be shortened; and the preset distance update condition can be pre-set. For example, in some application scenarios, the preset distance update condition can specifically be: the SDS loss corresponding to the three-dimensional representation model reaches the fifth loss threshold. For another example, in some application scenarios, the preset distance update condition can specifically be: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the first number threshold. For another example, in some application scenarios, the preset distance update condition can specifically be: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fifth number threshold. Based on this, it can be seen that between two adjacent distance shortening processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the fifth number threshold.

[0239] That is, when the fifth threshold value above is K, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Kth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the distance between the three-dimensional representation model and the image acquisition device needs to be shortened; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Kth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the distance between the three-dimensional representation model and the image acquisition device needs to be shortened; ... (and so on). Wherein, K is a positive integer, and K can be set according to the actual application scenario.

[0240] The image acquisition distance is used to represent the distance between an image acquisition device (eg, a camera) and a 3D representation model.

[0241] It should be noted that the present application does not limit the implementation method of shortening the distance between the three-dimensional representation model and the image acquisition device. For example, the increase process can be implemented using the following formula (4).

[0242] Image acquisition distance after update = image acquisition distance before update - preset distance step (4)

[0243] The preset distance step size refers to a single reduction amount set in advance for the image acquisition distance based on the application scenario.

[0244] In fact, in some application scenarios, in order to avoid the distance being too close, a lower limit of the distance can be set. Based on this, the present application also provides three possible implementation methods of the above preset distance update condition. In one possible implementation method, the preset distance update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the fifth loss threshold, and the above image acquisition distance is not lower than the preset distance threshold. In another possible implementation method, the preset distance update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the first number threshold, and the image acquisition distance is not lower than the preset distance threshold. In yet another possible implementation method, the preset distance update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fifth number threshold, and the image acquisition distance is not lower than the preset distance threshold.

[0245] Based on the relevant content of step 533 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model can be updated, but also the distance between the three-dimensional representation model and the image acquisition device can be shortened. In this way, as the three-dimensional representation model is iteratively updated, the image acquisition device gradually approaches the three-dimensional representation model. This is a gradual change process, so that the three-dimensional representation model can better learn the detailed information related to the object, so that the three-dimensional representation model can represent the object state more and more finely, which is conducive to improving the determination effect of the three-dimensional digital model.

[0246] It should be noted that, for step 53 above, if step 53 includes step 533 above, then since step 533 is used to update the image acquisition distance in the second posture description information above, so that the second posture description information changes after executing step 533, in order to ensure the model determination effect, it is necessary not only to re-acquire the second object description image, but also to re-acquire the second posture image, so that the "return to continue executing the image acquisition step and its subsequent steps" in step 53 can be specifically: return to continue executing step 51 above and its subsequent steps.

[0247] Based on the relevant content of the distance shortening operation above, it can be known that in one possible implementation method, the distance shortening operation is used to implement the above step of "if the preset distance update condition is met, then the image acquisition distance in the above second posture description information is updated so that the updated image acquisition distance is smaller than the image acquisition distance before the update", so that the gradual shortening of the image acquisition distance can be completed with the help of the distance shortening operation.

[0248] The focus range update operation is used to implement focus adjustment processing for the image acquisition device, so that the focus range update operation can be used to subsequently complete the process of taking a close-up of certain parts (such as the head) of the object described by the three-dimensional representation model (or three-dimensional parameterized model), thereby improving the refinement of the three-dimensional digital model. Based on this, the present application also provides a possible implementation of the update process in step 53 above. In this implementation, when the second posture description information above includes the image acquisition focus range, step 53 can at least include the following step 534.

[0249] Step 534: If the preset focus update condition is met, the image acquisition focus range in the second posture description information is updated so that the updated image acquisition focus range in the second posture description information is different from the image acquisition focus range before the update.

[0250] Among them, the preset focus update condition refers to the condition that needs to be met when the focus adjustment processing is performed on the image acquisition device; and the preset focus update condition can be pre-set. For example, in some application scenarios, the preset focus update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the sixth loss threshold. For another example, in some application scenarios, the preset focus update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the second quantity threshold. For another example, in some application scenarios, the preset focus update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the sixth number threshold. Based on this, it can be seen that between two adjacent focus adjustment processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the sixth number threshold.

[0251] That is, when the sixth threshold value above is Q, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Qth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the focus adjustment process needs to be performed on the image acquisition device; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Qth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the focus adjustment process needs to be performed on the image acquisition device; ... (and so on). Wherein, Q is a positive integer, and Q can be set according to the actual application scenario.

[0252] In fact, in some application scenarios, in order to avoid the focus range being too small, a lower limit of the focus range can be set. Based on this, the present application also provides three possible implementation methods of the above preset focus update condition. In one possible implementation method, the preset focus update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the sixth loss threshold, and the size of the focus range of the above image acquisition is not less than the preset range size threshold. In another possible implementation method, the preset focus update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the second number threshold, and the size of the focus range of the image acquisition is not less than the preset range size threshold. In yet another possible implementation method, the preset focus update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the sixth number threshold, and the size of the focus range of the image acquisition is not less than the preset range size threshold.

[0253] In addition, the present application does not limit the implementation method of the above focus adjustment processing. For ease of understanding, two examples are used below to illustrate.

[0254] Example 1: In one possible implementation, the focus adjustment process described above can be used to adjust the focus from the global range of an object to the local range of the object. Based on this, it can be seen that when the second posture description information described above refers to the information required to obtain an image of the second object, the image acquisition focus range before the update is used to describe the entire second object, and the image acquisition focus range after the update is used to describe the first part of the second object. The first part refers to any part of the second object that requires special attention (for example, the head, legs, etc.).

[0255] It can be seen that in some possible implementations, when the second posture description information mentioned above refers to information required for image acquisition of a second object, the image acquisition focus range before the update can be used to focus on the global range of the second object for image acquisition processing, and the image acquisition focus range after the update can be used to focus on the first local range of the second object for image acquisition processing. The image acquisition device that uses the global range of the second object as the image acquisition focus range is used to acquire a full image of the second object. The image acquisition device that uses the first local range of the second object as the image acquisition focus range is used to acquire a close-up image of the first portion of the second object.

[0256] Example 2, in another possible implementation, the focus adjustment process described above can be used to achieve adjustment from a local range of a certain object to a local range of another object. Based on this, it can be seen that when the second posture description information described above refers to the information required to obtain an image of the second object, the image acquisition focus range before the update is used to describe the second part of the second object, and the image acquisition focus range after the update is used to describe the third part of the second object, and the second part is different from the third part. The second part can refer to any part of the second object that needs to be focused on (for example, the head, legs, etc.); the third part can refer to any part of the second object that needs to be focused on but is different from the second part.

[0257] It can be seen that in some possible implementations, when the second posture description information mentioned above refers to information required for image acquisition of a second object, the image acquisition focus range before the update can be used to focus on a second local range of the second object for image acquisition processing, and the updated image acquisition focus range can be used to focus on a third local range of the second object for image acquisition processing, and the third local range is different from the second local range. The image acquisition device using the second local range of the second object as the image acquisition focus range is used to perform close-up image acquisition processing on the second part of the second object. The image acquisition device using the second local range of the second object as the image acquisition focus range is used to perform close-up image acquisition processing on the third part of the second object.

[0258] In addition, the present application does not limit the process of determining the image acquisition focus range in the above paragraph. For example, when the image acquisition focus range in the second posture description information above is used to describe the part to be focused (such as the head or legs, etc.), the process of determining the image acquisition focus range may include: first determining the position information of the part to be focused from the three-dimensional parametric model above, so that the position information can represent the position of the part to be focused in the three-dimensional parametric model; then, based on the position information, determining the image acquisition focus range, so that the image acquisition device can perform complete and clear image acquisition processing as close to the part to be focused as possible according to the image acquisition focus range, so that the image data obtained by rendering the three-dimensional parametric model using the image acquisition focus range can describe the part to be focused as finely as possible while ensuring integrity, and furthermore, the image data can represent a close-up effect of the part to be focused in the second object, so that the image data can subsequently be used to complete the refinement learning process of the three-dimensional representation model for the part to be focused, which is conducive to improving the refinement level of the three-dimensional representation model, and thus is conducive to improving the refinement level of the three-dimensional digital model.

[0259] Based on the relevant content of step 534 above, it can be seen that in some application scenarios, for the information update involved in certain rounds of processing, not only can the parameter information corresponding to each data unit in the three-dimensional representation model be updated, but the focus adjustment processing can also be performed on the image acquisition device, so that the image acquisition device can obtain some close-up image data for some parts of the second object, so that the three-dimensional representation model can subsequently use these close-up image data to complete the refinement learning process for these parts, so that the three-dimensional representation model can better describe the details of these parts, which is conducive to improving the refinement of the three-dimensional representation model, and thus is conducive to improving the refinement of the three-dimensional digital model.

[0260] It should be noted that, for step 53 above, if step 53 includes step 534 above, then since step 534 is used to update the image acquisition focus range in the second posture description information above, so that the second posture description information changes after executing step 534, in order to ensure the model determination effect, it is necessary not only to re-acquire the second object description image, but also to re-acquire the second posture image, so that the "return to continue executing the image acquisition step and its subsequent steps" in step 53 can be specifically: return to continue executing step 51 above and its subsequent steps.

[0261] Based on the relevant content of the focus range update operation above, it can be known that in one possible implementation, the focus range update operation is used to implement the above step of "if the preset focus update condition is met, then the image acquisition focus range is updated so that the updated image acquisition focus range is different from the image acquisition focus range before the update", so that the focus range update operation can be used to complete the learning process of some important close-up aspects (for example, the focus mode shown in Figure 6).

[0262] The model representation conversion operation is used to implement a model representation conversion process for the three-dimensional representation model, so that the model representation conversion operation can be used to complete a multi-stage update process (e.g., a coarse stage followed by a fine stage) of gradually increasing refinement of the three-dimensional representation model. Based on this, the present application also provides a possible implementation of the update process in step 53 above, which may include at least step 535 below.

[0263] Step 535: If the preset model conversion condition is met, the three-dimensional representation model is converted from the first model representation to the second model representation.

[0264] Among them, the preset model conversion condition refers to the condition that needs to be met when the model representation conversion process is required for the three-dimensional representation model; and the preset model conversion condition can be pre-set. For example, in some application scenarios, the preset model conversion condition can specifically be: the SDS loss corresponding to the three-dimensional representation model reaches the seventh loss threshold. For another example, in some application scenarios, the preset model conversion condition can specifically be: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the third quantity threshold. For another example, in some application scenarios, the preset model conversion condition can specifically be: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the seventh number threshold. Based on this, it can be seen that between two adjacent model representation conversion processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the seventh number threshold.

[0265] That is, when the seventh threshold value above is B, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Bth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the model representation conversion process needs to be performed for the three-dimensional representation model; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Bth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the model representation conversion process needs to be performed for the three-dimensional representation model; ... (and so on). Wherein, B is a positive integer, and B can be set according to the actual application scenario.

[0266] In fact, in some application scenarios, in order to avoid too many conversions, an upper limit on the number of conversions can be set. Based on this, the present application also provides three possible implementation methods of the above-mentioned preset model conversion conditions. In one possible implementation method, the preset model conversion condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the seventh loss threshold, and the number of conversions of the model expression of the three-dimensional representation model is lower than the eighth number threshold. In another possible implementation method, the preset model conversion condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the third number threshold, and the number of conversions of the model expression of the three-dimensional representation model is lower than the eighth number threshold. In yet another possible implementation method, the preset model conversion condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the seventh number threshold, and the number of conversions of the model expression of the three-dimensional representation model is lower than the eighth number threshold.

[0267] The first model representation refers to the model representation of the three-dimensional representation model before conversion; and this application does not limit the first model representation, for example, it can be a Nerf grid representation.

[0268] The second model representation refers to the model representation of the converted three-dimensional representation model; and the second model representation has a higher degree of refinement than the first model representation. In addition, this application does not limit the second model representation; for example, it can be a triangular mesh representation.

[0269] It should be noted that this application does not limit the implementation method of the above model representation conversion processing. For example, when the first model representation is a Nerf mesh representation and the second model representation is a triangular mesh representation, the model representation conversion processing can be specifically: first convert the Nerf mesh into a deformable tetrahedral mesh (Lin et al. 2022; Shen et al. 2021); and then determine the triangular mesh based on the deformable tetrahedral mesh.

[0270] Based on the relevant content of step 535 above, it can be seen that in some application scenarios, for the information update involved in certain rounds of processing, not only can the parameter information corresponding to each data unit in the three-dimensional representation model be updated, but the model representation conversion process can also be performed on the three-dimensional representation model. In this way, a multi-stage update process (for example, rough stage + fine stage) with gradually increasing fineness can be achieved for the three-dimensional representation model. This is a gradual change process, which is conducive to improving the determination effect of the three-dimensional digital model.

[0271] In addition, the present application does not limit the timing of determining the second stop condition in step 53 above. For example, the implementation method of determining the second stop condition in step 53 is similar to the implementation method of determining the second stop condition in step 43 above. For the sake of brevity, it will not be repeated here.

[0272] Based on the relevant content of steps 51 to 53 above, it can be seen that in some application scenarios, if multiple rounds of iterative update processes are required for the three-dimensional representation model, then for a certain round of processing in the multiple rounds of iterative update process, it may be necessary to update at least one information in this round of processing (for example, parameter information corresponding to each data unit in the three-dimensional representation model, the resolution of the three-dimensional representation model, the distribution of data units in the three-dimensional representation model, the distance between the three-dimensional representation model and the image acquisition device, the focus of the image acquisition device, and one or more information in the model representation method of the three-dimensional representation model). In this way, an iterative update process from coarse to fine can be realized for the three-dimensional representation model, which is conducive to determining a three-dimensional digital model with higher quality, so that the three-dimensional digital model can display the object as finely as possible under the premise of stability.

[0273] Based on the relevant content of the three-dimensional digital model determination process above, it can be seen that the present application provides a three-dimensional digital model determination method based on text and posture guidance, so that the three-dimensional digital model determination method can be used to create high-quality and stable three-dimensional digital models, avoiding the Janus Problem that often occurs when generating three-dimensional digital models based on two-dimensional diffusion models.

[0274] In addition, the three-dimensional digital model determination method provided in the present application is implemented by using density attitude conditioning and SDS as a guide, so that the three-dimensional digital model determined by using the three-dimensional digital model determination method has higher quality.

[0275] In addition, the 3D digital model determination method provided by this application helps to achieve high-resolution full / partial 3D digital model determination processing without the need for additional descriptive information. At the same time, it can also easily realize animation production using the parameters of the SMPL model without further training.

[0276] In addition, the three-dimensional digital model determination method provided in the present application improves the quality of the final determined three-dimensional digital model through some progressive information updating methods (for example, some progressive information updating methods shown in Figure 6, etc.). This method can ensure the implementation of the model refinement process from coarse to fine, so that the final determined three-dimensional digital model can present more detailed information (for example, the bending amplitude of fingers, certain body accessories, etc.).

[0277] It should be noted that the progressive information update method shown in the previous paragraph may include the following (1)-(3).

[0278] (1) The specific content related to the progressive grid can include the following ①-③.

[0279] ①The resolution of the Nerf grid gradually increases with the iterative update process of the three-dimensional representation model.

[0280] ② During the iterative update of the 3D representation model, the density threshold is repeatedly used to filter the scene, and a bounding box (bbox) is used to tightly surround the area. This is used to reduce the number of grids used and increase the generation speed.

[0281] ③ Progressive camera distance: The camera distance is gradually shortened as the 3D representation model is iteratively updated.

[0282] (2) The content related to the close-up camera position may include the following ④-⑥.

[0283] ④ To generate finer details in certain areas of the body that require special attention, this application introduces focus in both the coarse and fine stages. The coarse stage refers to a multi-round iterative update process using the Nerf mesh as the update target, while the fine stage refers to a multi-round iterative update process using the triangular mesh as the update target.

[0284] ⑤ Due to the SMPL prior, this application can first calculate the position of the original body part under a certain posture description information; then by moving the camera close to the body part, the loss calculation can be performed with high resolution in a very small focus area.

[0285] ⑥ Since the ControlNet built based on DensePose provided by this application has high stability, body parts can be generated without using additional computing resources. This allows this application to promote detailed learning of high-quality three-dimensional digital models with the help of focus mode (close-up camera position).

[0286] (3) The content related to triangular mesh refinement can include the following ⑦-⑧.

[0287] ⑦ Compared with Nerf mesh, triangular mesh is used more frequently in application scenarios, so this application can convert the Nerf mesh determined in the coarse stage into a triangular mesh, and continue the iterative update processing of the fine stage based on the triangular mesh.

[0288] ⑧Integrate deformable tetrahedral meshes (Lin et al. 2022; Shen et al. 2021) to learn a textured three-dimensional representation model; and this application uses the trained explicit Nerf mesh as the initialization of the three-dimensional representation model, and optimizes the three-dimensional representation model by backpropagation using DensePose conditional SDS gradients.

[0289] Furthermore, this application does not limit the execution entity of the above-mentioned three-dimensional digital model determination method. For example, any implementation of the three-dimensional digital model determination method provided in the embodiments of this application can be applied to a device with data processing capabilities, such as a terminal device or a server. For another example, any implementation of the three-dimensional digital model determination method provided in the embodiments of this application can also be implemented through a data communication process between different devices (e.g., a terminal device and a server, two terminal devices, or two servers).

[0290] In fact, in some application scenarios (such as virtual try-on scenarios), the determined three-dimensional digital model can be rendered and displayed according to certain posture description information to meet the user's usage requirements for the three-dimensional digital model in the application scenario. Based on this, the present application also provides a three-dimensional digital model display method, as shown in Figure 7, which may include the following S701-S703. Figure 7 is a flowchart of a three-dimensional digital model display method provided in an embodiment of the present application.

[0291] S701: Acquire fourth posture description information.

[0292] The fourth posture description information refers to the posture description information required when using a predetermined three-dimensional digital model. It should be noted that the implementation of the fourth posture description information is similar to the implementation of the first posture description information above, and for the sake of brevity, it is not repeated here.

[0293] In addition, the present application does not limit the method for obtaining the fourth posture description information. For example, it can be specifically: the user of the terminal device inputs the fourth posture description information via an input device in the terminal device. It should be noted that the present application does not limit the input method of the fourth posture description information.

[0294] S702: Rendering a predetermined three-dimensional digital model using the fourth posture description information, where the three-dimensional digital model is determined using any implementation of the three-dimensional digital model determination method provided in this application.

[0295] In the present application, after obtaining the fourth posture description information, the fourth posture description information can be used to render a predetermined three-dimensional digital model to obtain a rendered three-dimensional digital model, so that the rendered three-dimensional digital model can display the object according to the object state described by the fourth posture description information, thereby enabling the rendered three-dimensional digital model to better meet the user's model perspective display requirements, which is conducive to improving the user experience.

[0296] S703: Display a rendering result, where the rendering result satisfies the constraint described by the fourth posture description information.

[0297] The rendering result is obtained by rendering the three-dimensional digital model using the fourth posture description information, so that the rendering result satisfies the constraints described by the fourth posture description information.

[0298] In addition, this application does not limit the representation method of the above rendering results. For example, in some application scenarios, the rendering results can be represented by image data. In other application scenarios, the rendering results can be represented by a three-dimensional digital model.

[0299] Based on the relevant contents of S701 to S703 above, it can be seen that after the three-dimensional digital model is pre-determined, the three-dimensional digital model can be rendered according to the fourth posture description information provided by the user, so that the three-dimensional digital model can be displayed according to the object state described by the fourth posture description information, thereby meeting the user's model perspective display requirements. In particular, because the three-dimensional digital model can finely display the object state from various perspectives, it can better meet the user's model perspective display requirements, thereby improving the user experience.

[0300] In addition, this application does not limit the execution subject of the above-mentioned three-dimensional digital model display method. For example, any implementation of the three-dimensional digital model display method provided in the embodiments of this application can be applied to a device with data processing capabilities, such as a terminal device or a server. For another example, any implementation of the three-dimensional digital model display method provided in the embodiments of this application can also be implemented through a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0301] Based on the image generation model construction method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation model construction device, which is explained and illustrated below in conjunction with Figure 8. Figure 8 is a structural schematic diagram of an image generation model construction device provided in the embodiment of the present application. It should be noted that for the technical details of the image generation model construction device provided in the embodiment of the present application, please refer to the relevant content of the image generation model construction method above.

[0302] As shown in FIG8 , the image generation model construction device 800 provided in an embodiment of the present application includes:

[0303] A first acquiring unit 801 is configured to acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image;

[0304] The model construction unit 802 is used to construct an image generation model using the first object description image, the first description text and the first posture image. The image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0305] In one possible implementation, the process of determining the first posture image includes: determining first posture description information from the first object description image; and rendering a three-dimensional parametric model using the first posture description information to obtain a first posture image, where the three-dimensional parametric model is a three-dimensional digital model.

[0306] In a possible implementation, the first object description image is used to describe at least one part of the first object; the first posture image carries part labeling information corresponding to each part, and the part labeling information corresponding to different parts is different.

[0307] In a possible implementation manner, the colors of regions where different parts of the first posture image are located are different.

[0308] In one possible implementation, the model building unit 802 is specifically used to: input the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model; update the preset diffusion model based on the model output data, and continue to execute the steps of obtaining the first object description image, the first description text corresponding to the first object description image, and the first posture image corresponding to the first object description image until the first stop condition is reached, and then determine the image generation model based on the preset diffusion model.

[0309] Based on the relevant content of the above-mentioned image generation model construction device 800, it can be known that for the image generation model construction device 800 provided in the embodiment of the present application, the first object description image (for example, image data in a certain training image set), the first description text corresponding to the first object description image (for example, an image description text similar to "There is a stone bench by the river"), and the first posture image corresponding to the first object description image (for example, a dense posture rendering image, etc.) are first obtained; then the first object description image, the first description text and the first posture image are used to construct an image generation model, so that the constructed image generation model can generate relatively high-quality image data based on the first description text and the first posture image, so that the image generation model can achieve the purpose of generating image data based on text information and posture information, which is beneficial to improving the image generation effect of the image generation model. Among them, because the above-mentioned first posture image is determined based on the first object description image, the first posture image can better represent the state of the object in the first object description image (for example, which parts exist and what postures each part is in, etc.), so that the image data generated with the first posture image as a guiding condition is more in line with the natural posture constraints corresponding to different parts of the object (for example, the head, etc.), and thus the image data generated with the first posture image as a guiding condition has higher quality, which can effectively avoid the problems caused when generating image data based only on text information, thereby helping to improve the image generation effect.

[0310] Based on the image generation method provided in the embodiments of this application, the embodiments of this application also provide an image generation device, which will be explained and illustrated below in conjunction with Figure 9. Figure 9 is a schematic structural diagram of an image generation device provided in the embodiments of this application. It should be noted that for the technical details of the image generation device provided in the embodiments of this application, please refer to the relevant content of the image generation method above.

[0311] As shown in FIG9 , an image generation device 900 provided in an embodiment of the present application includes:

[0312] The second acquiring unit 901 is configured to acquire a third description text and third posture description information corresponding to the third description text;

[0313] A second rendering unit 902 is configured to render the three-dimensional parameterized model using the third posture description information to obtain a third posture image;

[0314] The image generation unit 903 is used to obtain a generated image using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using any implementation of the image generation model construction method provided in this application.

[0315] Based on the relevant content of the above-mentioned image generation device 900, it can be known that for the image generation device 900 provided in the embodiment of the present application, after obtaining the constructed image generation model, the image generation model can be used to complete the task of performing image generation processing based on text + additional conditions. Among them, because the image generation model performs image generation processing based on text + posture image, so that the image data generated by using the image generation model conforms to both the image description content carried by the text and the object state described by the posture image (for example, the natural posture constraints of some body parts), it can effectively avoid the defects caused by the generated image data not conforming to the natural posture constraints of some body parts (for example, generating inferior image data that does not conform to natural laws), which is conducive to improving the image generation effect.

[0316] Based on the three-dimensional digital model determination method provided in the embodiments of this application, the embodiments of this application also provide a three-dimensional digital model determination device, which is explained and illustrated below in conjunction with Figure 10. Figure 10 is a schematic structural diagram of the three-dimensional digital model determination device provided in the embodiments of this application. It should be noted that for the technical details of the three-dimensional digital model determination device provided in the embodiments of this application, please refer to the relevant content of the three-dimensional digital model determination method above.

[0317] As shown in FIG10 , the three-dimensional digital model determination device 1000 provided in an embodiment of the present application includes:

[0318] A first rendering unit 1001 is configured to render the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and to render the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0319] A data determination unit 1002 is configured to input the second object description image, the second posture image, and a second description text corresponding to the second object description image into a pre-built image generation model to obtain model output data of the image generation model; the image generation model is configured to generate image data based on the second posture image and the second description text, the image data satisfying the constraints described by the second description text and the constraints represented by the second posture image;

[0320] An information updating unit 1003 is configured to update the three-dimensional representation model according to the model output data;

[0321] The model determining unit 1004 is configured to determine a three-dimensional digital model according to the three-dimensional representation model.

[0322] In one possible implementation, the image generation model is constructed using any implementation of the image generation model construction method provided in this application.

[0323] In one possible implementation, the image generation model is a diffusion model, and the model output data includes predicted noise;

[0324] The information updating unit 1003 is specifically used to: determine the SDS loss corresponding to the three-dimensional representation model based on the predicted noise, the noise label corresponding to the second object description image, and the score distillation sampling SDS loss function; the noise label refers to the noise data actually used in the process of determining the model output data; and update the three-dimensional representation model based on the SDS loss.

[0325] In a possible implementation manner, the information updating unit 1003 is specifically configured to update parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data.

[0326] In one possible implementation, the information updating unit 1003 is further configured to: after updating the three-dimensional representation model according to the model output data, return to the first rendering unit 1001 to continue executing the image acquisition step, wherein the image acquisition step at least includes the step of rendering the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0327] The model determination unit 1004 is specifically configured to determine a three-dimensional digital model according to the three-dimensional representation model until a second stop condition is reached.

[0328] In a possible implementation, the information updating unit 1003 is further configured to update the resolution of the 3D representation model if a preset resolution update condition is met, so that the resolution of the updated 3D representation model is higher than the resolution of the 3D representation model before the update.

[0329] In a possible implementation, the information updating unit 1003 is further configured to search for and delete a target data unit from the three-dimensional representation model if a preset data unit filtering condition is met, and a density parameter corresponding to the target data unit is lower than a preset density threshold.

[0330] In a possible implementation manner, the second posture description information includes an image acquisition distance;

[0331] The information updating unit 1003 is further configured to: if a preset distance update condition is satisfied, update the image acquisition distance in the second posture description information so that the updated image acquisition distance is smaller than the image acquisition distance before the update;

[0332] The image acquisition step includes: rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

[0333] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0334] The information updating unit 1003 is further configured to: if a preset focus update condition is satisfied, update the image acquisition focus range so that the updated image acquisition focus range is different from the image acquisition focus range before the update;

[0335] The image acquisition step includes: determining a second posture image from the three-dimensional parameterized model rendered using the second posture description information, and determining a second object description image from the three-dimensional representation model rendered using the second posture description information.

[0336] In one possible implementation, the second posture description information refers to information required to be used as a basis for acquiring an image of the second object; the image acquisition focus range before the update is used to describe the entire second object, and the image acquisition focus range after the update is used to describe the first part of the second object; or, the image acquisition focus range before the update is used to describe the second part of the second object, and the image acquisition focus range after the update is used to describe the third part of the second object.

[0337] In a possible implementation manner, the information updating unit 1003 is further configured to: if a preset model conversion condition is met, convert the three-dimensional representation model from the first model representation mode to the second model representation mode.

[0338] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0339] If the image acquisition focus range is used to describe the portion to be focused, the process of determining the image acquisition focus range includes: determining the position information of the portion to be focused from the three-dimensional parametric model; and determining the image acquisition focus range based on the position information, so that the image data obtained by rendering the three-dimensional parametric model using the image acquisition focus range is used to describe the portion to be focused.

[0340] Based on the relevant content of the above-mentioned three-dimensional digital model determination device 1000, it can be known that for the three-dimensional digital model determination device 1000 provided in the embodiment of the present application, after obtaining the constructed image generation model, because the image generation model can perform image generation processing for any perspective (for example, a certain camera focus + some body parts in a certain posture, etc.), so that the image data at any perspective generated based on the image generation model conforms to the natural posture constraints corresponding to the perspective, so that the three-dimensional digital model determined based on the image generation model can more accurately represent the state of the object at different perspectives, thereby making the three-dimensional digital model have better performance, which is conducive to improving the determination effect of the three-dimensional digital model.

[0341] Based on the three-dimensional digital model display method provided in the embodiments of this application, the embodiments of this application also provide a three-dimensional digital model display device, which is explained and illustrated below in conjunction with Figure 11. Figure 11 is a schematic structural diagram of a three-dimensional digital model display device provided in the embodiments of this application. It should be noted that for the technical details of the three-dimensional digital model display device provided in the embodiments of this application, please refer to the relevant content of the three-dimensional digital model display method above.

[0342] As shown in FIG11 , a three-dimensional digital model display device 1100 provided in an embodiment of the present application includes:

[0343] The third acquiring unit 1101 is configured to acquire fourth posture description information;

[0344] A model adjustment unit 1102 is configured to render a predetermined three-dimensional digital model using the fourth posture description information, where the three-dimensional digital model is determined using any embodiment of the three-dimensional digital model determination method provided in this application;

[0345] The model display unit 1103 is configured to display a rendering result, where the rendering result satisfies the constraint described by the fourth posture description information.

[0346] Based on the above-mentioned relevant content of the three-dimensional digital model display device 1100, it can be seen that for the three-dimensional digital model display device 1100 provided in the embodiment of the present application, after pre-determining the three-dimensional digital model, the three-dimensional digital model can be rendered according to the fourth posture description information provided by the user, so that the three-dimensional digital model can be displayed according to the object state described by the fourth posture description information, thereby meeting the user's model perspective display requirements. In particular, because the three-dimensional digital model can finely display the object state from various perspectives, the three-dimensional digital model can better meet the user's model perspective display requirements, thereby improving the user experience.

[0347] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any embodiment of the image generation model construction method provided in this application, or executes any embodiment of the three-dimensional digital model determination method provided in this application, or executes any embodiment of the image generation method provided in this application, or executes any embodiment of the three-dimensional digital model display method provided in this application.

[0348] Referring to FIG12 , a schematic diagram of the structure of an electronic device 1200 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG12 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0349] As shown in FIG12 , the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0350] Typically, the following devices may be connected to the I / O interface 1205: an input device 1206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1208 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although FIG12 shows the electronic device 1200 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may be implemented or present instead.

[0351] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1209, or installed from the storage device 1208, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0352] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0353] An embodiment of the present application also provides a computer-readable medium, which stores instructions or computer programs. When the instructions or computer programs are executed on a device, the device executes any embodiment of the image generation model construction method provided in the present application, or executes any embodiment of the three-dimensional digital model determination method provided in the present application, or executes any embodiment of the image generation method provided in the present application, or executes any embodiment of the three-dimensional digital model display method provided in the present application.

[0354] Compared with the prior art, this application has at least the following advantages:

[0355] In the technical solution provided in the present application, a first object description image (for example, image data in a certain training image set), a first description text corresponding to the first object description image (for example, an image description text similar to "There is a stone bench by the river"), and a first posture image corresponding to the first object description image (for example, a dense posture rendering image, etc.) are first obtained; then, the first object description image, the first description text and the first posture image are used to construct an image generation model, so that the constructed image generation model can generate image data of relatively high quality based on the first description text and the first posture image, thereby enabling the image generation model to achieve the purpose of generating image data based on text information and posture information, which is conducive to improving the image generation effect of the image generation model.

[0356] Among them, because the above-mentioned first posture image is determined based on the first object description image, the first posture image can better represent the state of the object in the first object description image (for example, which parts exist and what postures each part is in, etc.), so that the image data generated with the first posture image as a guiding condition is more in line with the natural posture constraints corresponding to different parts of the object (for example, the head, etc.), and thus the image data generated with the first posture image as a guiding condition has higher quality, which can effectively avoid the problems caused when generating image data based only on text information, thereby helping to improve the image generation effect.

[0357] In addition, because the image data generated by the image generation model conforms to the natural posture constraints corresponding to different parts of the object, the three-dimensional digital model (for example, a three-dimensional digital human, a three-dimensional digital creature, etc.) determined with the help of the image generation model can also conform to these natural posture constraints, thereby effectively overcoming the defects of related three-dimensional digital model determination schemes (for example, defects such as the Janus Problem), and thus helping to improve the determination effect of the three-dimensional digital model.

[0358] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0359] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0360] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0361] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0362] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0363] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0364] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0365] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0366] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0367] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0368] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0369] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0370] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0371] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an image generation model, wherein the method comprises: Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image; An image generation model is constructed using the first object description image, the first description text and the first posture image. The image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

2. The method according to claim 1, wherein the process of determining the first posture image comprises: determining first posture description information from the first object description image; The three-dimensional parameterized model is rendered using the first posture description information to obtain a first posture image, and the three-dimensional parameterized model is a three-dimensional digital model.

3. The method according to claim 1, wherein the first object description image is used to describe at least a portion of a first object; The first posture image carries part labeling information corresponding to each part, and the part labeling information corresponding to different parts is different. The method according to claim 3 , wherein the colors of the regions where different parts of the first posture image are located are different.

5. The method according to any one of claims 1 to 4, wherein the process of constructing the image generation model comprises: Inputting the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model; According to the model output data, the preset diffusion model is updated, and the steps of acquiring a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image are continued until a first stop condition is reached, and the image generation model is determined according to the preset diffusion model.

6. A method for determining a three-dimensional digital model, wherein the method comprises: Rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image; Inputting the second object description image, the second posture image and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data according to the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image; updating the three-dimensional representation model according to the model output data; A three-dimensional digital model is determined according to the three-dimensional representation model.

7. The method according to claim 6, wherein the image generation model is constructed using the image generation model construction method according to any one of claims 1-5.

8. The method according to claim 6, wherein the image generation model is a diffusion model, and the model output data includes predicted noise; The updating of the three-dimensional representation model according to the model output data comprises: Determining the SDS loss corresponding to the three-dimensional representation model according to the predicted noise, the noise label corresponding to the second object description image, and the score distillation sampling SDS loss function; the noise label refers to the noise data actually used in the process of determining the output data of the model; The three-dimensional representation model is updated according to the SDS loss.

9. The method according to claim 6, wherein updating the three-dimensional representation model according to the model output data comprises: According to the model output data, parameter information corresponding to each data unit in the three-dimensional representation model is updated.

10. The method according to any one of claims 6 to 9, wherein after updating the three-dimensional representation model according to the model output data, the method further comprises: Continue to execute the image acquisition step, the image acquisition step at least includes the step of rendering the three-dimensional representation model using the second posture description information to obtain a second object description image; Determining a three-dimensional digital model according to the three-dimensional representation model includes: When the second stop condition is reached, a three-dimensional digital model is determined according to the three-dimensional representation model.

11. The method according to claim 10, wherein before continuing to perform the image acquisition step, the method further comprises: If the preset resolution update condition is met, the resolution of the three-dimensional representation model is updated so that the resolution of the updated three-dimensional representation model is higher than the resolution of the three-dimensional representation model before the update.

12. The method according to claim 10, wherein before continuing to perform the image acquisition step, the method further comprises: Searching and deleting a target data unit from the three-dimensional representation model, wherein a density parameter corresponding to the target data unit is lower than a preset density threshold; or, If the preset data unit filtering condition is met, the target data unit is searched and deleted from the three-dimensional representation model.

13. The method according to claim 10, wherein the second posture description information includes an image acquisition distance; Before continuing to perform the image acquisition step, the method further includes: If the preset distance update condition is met, updating the image acquisition distance in the second posture description information so that the updated image acquisition distance is smaller than the image acquisition distance before the update; The step of continuing to perform the image acquisition step comprises: Continue to perform the steps of rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

14. The method according to claim 10, wherein the second posture description information includes an image acquisition focus range; Before continuing to perform the image acquisition step, the method further includes: If the preset focus update condition is met, updating the image acquisition focus range so that the updated image acquisition focus range is different from the image acquisition focus range before the update; The step of continuing to perform the image acquisition step comprises: Continue to perform the steps of rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

15. The method according to claim 14, wherein the second posture description information refers to information required to be based on when acquiring the second object; The image acquisition focus range before updating is used to describe the whole of the second object, and the image acquisition focus range after updating is used to describe the first part of the second object; or, the image acquisition focus range before updating is used to describe the second part of the second object, and the image acquisition focus range after updating is used to describe the third part of the second object.

16. The method according to claim 10, wherein before continuing to perform the image acquisition step, the method further comprises: If the preset model conversion condition is met, the three-dimensional representation model is converted from the first model representation mode to the second model representation mode.

17. The method according to claim 6, wherein the second posture description information includes an image acquisition focus range; If the image acquisition focus range is used to describe the portion to be focused, the process of determining the image acquisition focus range includes: Determining position information of the portion to be focused from the three-dimensional parameterized model; The image acquisition focus range is determined according to the position information, so that the image data obtained by rendering the three-dimensional parameterized model using the image acquisition focus range is used to describe the part to be focused.

18. A method for generating an image, comprising: Acquire a third description text and third posture description information corresponding to the third description text; Rendering the three-dimensional parameterized model using the third posture description information to obtain a third posture image; A generated image is obtained using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using the image generation model construction method described in any one of claims 1-5.

19. An image generation model construction device, comprising: A first acquisition unit, configured to acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image; A model building unit is used to build an image generation model using the first object description image, the first description text and the first posture image, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

20. A three-dimensional digital model determination device, comprising: A first rendering unit, configured to render the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and to render the three-dimensional representation model using the second posture description information to obtain a second object description image; The data determining unit is configured to input the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain a model of the image generation model. The image generation model is used to generate image data according to the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image; An information updating unit, configured to update the three-dimensional representation model according to the model output data; The model determination unit is used to determine the three-dimensional digital model according to the three-dimensional representation model.

21. An image generating device, comprising: A second acquisition unit, used to acquire a third description text and third posture description information corresponding to the third description text; A second rendering unit, configured to render the three-dimensional parameterized model using the third posture description information to obtain a third posture image; An image generation unit is used to obtain a generated image using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using the image generation model construction method described in any one of claims 1-5.

22. An electronic device, wherein the device comprises: Processor and memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 18.

23. A computer-readable medium, wherein the computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 18.

Images