Image generation model construction method and three-dimensional digital model determination method

Abstract

The invention discloses an image generation model construction method and a three-dimensional digital model determination method. The method comprises the following steps: firstly, obtaining a first object description image, a first description text corresponding to the first object description image, and a first attitude image corresponding to the first object description image; and constructing an image generation model by using the first object description image, the first description text and the first attitude image, so that the constructed image generation model can generate image data with relatively high quality according to the first description text and the first attitude image. Therefore, the image generation model can achieve the purpose of generating the image data according to the text information and the posture information, the image generation effect of the image generation model is improved, and a three-dimensional digital model determined by means of the image generation model can also conform to natural posture constraints. And the determination effect of the three-dimensional digital model can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image generation model construction method, a three-dimensional digital model determination method, an image generation method, a three-dimensional digital model display method, a device, an electronic device, and a computer-readable medium. Background Art

[0002] In some application scenarios (for example, image generation scenarios, image style conversion scenarios, etc.), it may be necessary to pre-build a model so that the model can generate image data that meets the requirements of the application scenario (for example, image data with a certain style, etc.). This makes how to build the model a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides an image generation model construction method, a three-dimensional digital model determination method, an image generation method, a three-dimensional digital model display method, a device, an electronic device, and a computer-readable medium.

[0004] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0005] The present application provides a method for constructing an image generation model, the method comprising:

[0006] Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image;

[0007] An image generation model is constructed using the first object description image, the first description text and the first posture image. The image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0008] In one possible implementation, the process of determining the first posture image includes: determining first posture description information from the first object description image; rendering a three-dimensional parameterized model using the first posture description information to obtain a first posture image, wherein the three-dimensional parameterized model is a three-dimensional digital model.

[0009] In a possible implementation manner, the first object description image is used to describe at least one part of the first object; the first posture image carries part labeling information corresponding to each part, and the part labeling information corresponding to different parts is different.

[0010] In a possible implementation manner, the colors of regions where different parts of the first posture image are located are different.

[0011] In one possible implementation, the process of constructing the image generation model includes: inputting the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model; updating the preset diffusion model according to the model output data, and continuing to execute the steps of obtaining the first object description image, the first description text corresponding to the first object description image, and the first posture image corresponding to the first object description image, until the first stop condition is reached, and determining the image generation model according to the preset diffusion model.

[0012] The present application provides a three-dimensional digital model determination method, the method comprising:

[0013] Rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0014] Inputting the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data according to the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image;

[0015] updating the three-dimensional representation model according to the model output data;

[0016] A three-dimensional digital model is determined according to the three-dimensional representation model.

[0017] In one possible implementation, the image generation model is constructed using the image generation model construction method provided in the present application.

[0018] In a possible implementation manner, the image generation model is a diffusion model, and the model output data includes prediction noise;

[0019] The updating of the three-dimensional representation model according to the model output data includes: determining the SDS loss corresponding to the three-dimensional representation model according to the predicted noise, the noise label corresponding to the second object description image, and the score distillation sampling SDS loss function; the noise label refers to the noise data actually used in the process of determining the model output data; and updating the three-dimensional representation model according to the SDS loss.

[0020] In a possible implementation manner, updating the three-dimensional representation model according to the model output data includes: updating parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data.

[0021] In a possible implementation manner, after updating the three-dimensional representation model according to the model output data, the method further includes:

[0022] Continue to execute the image acquisition step, the image acquisition step at least includes the step of rendering the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0023] Determining a three-dimensional digital model according to the three-dimensional representation model includes:

[0024] When the second stop condition is reached, a three-dimensional digital model is determined according to the three-dimensional representation model.

[0025] In a possible implementation manner, before continuing to perform the image acquisition step, the method further includes:

[0026] If the preset resolution update condition is met, the resolution of the three-dimensional representation model is updated so that the resolution of the updated three-dimensional representation model is higher than the resolution of the three-dimensional representation model before the update.

[0027] In a possible implementation manner, before continuing to perform the image acquisition step, the method further includes:

[0028] If the preset data unit filtering condition is met, a target data unit is searched and deleted from the three-dimensional representation model, and a density parameter corresponding to the target data unit is lower than a preset density threshold.

[0029] In a possible implementation manner, the second posture description information includes an image acquisition distance;

[0030] Before continuing to perform the image acquisition step, the method further includes:

[0031] If the preset distance update condition is met, updating the image acquisition distance in the second posture description information so that the updated image acquisition distance is smaller than the image acquisition distance before the update;

[0032] The step of continuing to perform the image acquisition step comprises:

[0033] Continue to perform the steps of rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

[0034] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0035] Before continuing to perform the image acquisition step, the method further includes:

[0036] If the preset focus update condition is met, updating the image acquisition focus range so that the updated image acquisition focus range is different from the image acquisition focus range before the update;

[0037] The step of continuing to perform the image acquisition step comprises:

[0038] The steps of determining a second posture image from the three-dimensional parameterized model rendered using the second posture description information and determining a second object description image from the three-dimensional representation model rendered using the second posture description information are continued.

[0039] In one possible implementation, the second posture description information refers to information required to be used as a basis for acquiring images of the second object; the image acquisition focus range before the update is used to describe the second object as a whole, and the image acquisition focus range after the update is used to describe the first part of the second object; or, the image acquisition focus range before the update is used to describe the second part of the second object, and the image acquisition focus range after the update is used to describe the third part of the second object.

[0040] In a possible implementation manner, before continuing to perform the image acquisition step, the method further includes:

[0041] If the preset model conversion condition is met, the three-dimensional representation model is converted from the first model representation mode to the second model representation mode.

[0042] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0043] If the image acquisition focus range is used to describe the part to be focused, the process of determining the image acquisition focus range includes: determining the position information of the part to be focused from the three-dimensional parameterized model; determining the image acquisition focus range based on the position information, so that the image data obtained by rendering the three-dimensional parameterized model using the image acquisition focus range is used to describe the part to be focused.

[0044] The present application provides an image generation method, the method comprising:

[0045] Acquire a third description text and third posture description information corresponding to the third description text;

[0046] Rendering the three-dimensional parameterized model using the third posture description information to obtain a third posture image;

[0047] A generated image is obtained by using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed by using the image generation model construction method provided in the present application.

[0048] The present application provides a three-dimensional digital model display method, the method comprising:

[0049] Obtaining fourth posture description information;

[0050] Rendering a predetermined three-dimensional digital model using the fourth posture description information, wherein the three-dimensional digital model is determined using the three-dimensional digital model determination method provided in the present application;

[0051] A rendering result is displayed, where the rendering result satisfies the constraint described by the fourth posture description information.

[0052] The present application provides an image generation model construction device, comprising:

[0053] A first acquisition unit, configured to acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image;

[0054] A model building unit is used to build an image generation model using the first object description image, the first description text and the first posture image, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0055] The present application provides a three-dimensional digital model determination device, comprising:

[0056] A first rendering unit, configured to render the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and to render the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0057] a data determination unit, configured to input the second object description image, the second posture image, and a second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is configured to generate image data according to the second posture image and the second description text, the image data satisfying the constraints described by the second description text and the constraints represented by the second posture image;

[0058] An information updating unit, configured to update the three-dimensional representation model according to the model output data;

[0059] The model determination unit is used to determine the three-dimensional digital model according to the three-dimensional representation model.

[0060] The present application provides an image generation device, comprising:

[0061] A second acquisition unit, used to acquire a third description text and third posture description information corresponding to the third description text;

[0062] A second rendering unit, configured to render the three-dimensional parameterized model using the third posture description information to obtain a third posture image;

[0063] An image generation unit is used to obtain a generated image using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using the image generation model construction method provided in the present application.

[0064] The present application provides a three-dimensional digital model display device, comprising:

[0065] A third acquiring unit, used to acquire fourth posture description information;

[0066] A model adjustment unit, configured to render a predetermined three-dimensional digital model using the fourth posture description information, wherein the three-dimensional digital model is determined using the three-dimensional digital model determination method provided in the present application;

[0067] A model display unit is used to display a rendering result, where the rendering result satisfies the constraints described by the fourth posture description information.

[0068] The present application provides an electronic device, the device comprising: a processor and a memory;

[0069] The memory is used to store instructions or computer programs;

[0070] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the image generation model construction method provided in the present application, or executes the three-dimensional digital model determination method provided in the present application, or executes the image generation method provided in the present application, or executes the three-dimensional digital model display method provided in the present application.

[0071] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the image generation model construction method provided in the present application, or executes the three-dimensional digital model determination method provided in the present application, or executes the image generation method provided in the present application, or executes the three-dimensional digital model display method provided in the present application.

[0072] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the image generation model construction method provided by the present application, or program code for executing the three-dimensional digital model determination method provided by the present application, or program code for executing the image generation method provided by the present application, or program code for executing the three-dimensional digital model display method provided by the present application.

[0073] Compared with the prior art, this application has at least the following advantages:

[0074] In the technical solution provided in the present application, a first object description image (for example, image data in a certain training image set), a first description text corresponding to the first object description image (for example, an image description text similar to "There is a stone bench by the river"), and a first posture image corresponding to the first object description image (for example, a dense posture rendering image, etc.) are first obtained; then, an image generation model is constructed using the first object description image, the first description text, and the first posture image, so that the constructed image generation model can generate image data of relatively high quality based on the first description text and the first posture image, so that the image generation model can achieve the purpose of generating image data based on text information and posture information, which is beneficial to improving the image generation effect of the image generation model.

[0075] Among them, because the above-mentioned first posture image is determined based on the first object description image, the first posture image can better express the state of the object in the first object description image (for example, which parts exist and what postures of each part are, etc.), so that the image data generated with the first posture image as a guiding condition is more in line with the natural posture constraints corresponding to different parts of the object (for example, the head, etc.), and thus the image data generated with the first posture image as a guiding condition has higher quality, which can effectively avoid the problems caused when generating image data only based on text information, thereby helping to improve the image generation effect.

[0076] In addition, because the image data generated by the image generation model conforms to the natural posture constraints corresponding to different parts of the object, the three-dimensional digital model (for example, three-dimensional digital human, three-dimensional digital creature, etc.) determined with the help of the image generation model can also conform to these natural posture constraints, thereby effectively overcoming the defects of related three-dimensional digital model determination schemes (for example, Janus Problem, etc.), and thus helping to improve the determination effect of the three-dimensional digital model. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 A flowchart of a method for constructing an image generation model provided in an embodiment of the present application;

[0079] Figure 2 A schematic diagram of generated image data provided by an embodiment of the present application;

[0080] Figure 3 A flowchart of an image generation method provided in an embodiment of the present application;

[0081] Figure 4 A flowchart of a method for determining a three-dimensional digital model provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of a process for determining a three-dimensional digital model provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of a progressive high-resolution model generation process provided in an embodiment of the present application;

[0084] Figure 7 A flowchart of a three-dimensional digital model display method provided in an embodiment of the present application;

[0085] Figure 8 A schematic diagram of the structure of an image generation model construction device provided in an embodiment of the present application;

[0086] Fig. 9 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0087] Fig.10 A schematic diagram of the structure of a three-dimensional digital model determination device provided in an embodiment of the present application;

[0088] Fig.11 A schematic diagram of the structure of a three-dimensional digital model display device provided in an embodiment of the present application;

[0089] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0091] In order to better understand the technical solution provided by the present application, the image generation model construction method provided by the present application is described below with reference to some drawings. Figure 1 As shown, the image generation model construction method provided in the embodiment of the present application includes the following S101-S102. Figure 1 A flowchart of a method for constructing an image generation model provided in an embodiment of the present application.

[0092] S101: Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image.

[0093] Among them, the first object description image refers to the image data required to be used when constructing the image generation model below; and the first object description image is used to describe at least one part of the first object. The first object refers to the object described by the first object description image, and the present application does not limit the first object. For example, the first object can be a person, an animal, or an object. It should be noted that the present application does not limit the at least one part. For example, when the first object refers to a person or an animal, the at least one part of the first object can include at least one body part of the first object (for example, all body parts, or only the head, or only the legs, etc.). For another example, when the first object refers to an object (for example, a car, a building, etc.), the at least one part of the first object can include at least one component of the first object.

[0094] In addition, the present application does not limit the method of obtaining the first object description image mentioned above. For example, it can be specifically: using image data in a preset training image set as the first object description image. It should be noted that the present application does not limit the implementation method of the training image set. For example, it can be implemented using the DeepFashion (Liu et al, 2016) dataset.

[0095] The first description text refers to the text information required to be used when the first object description image is used for model training, so that the first description text can describe what the image data generated by the model looks like. For example, the first description text can be implemented using an image description text such as "There is a stone bench by the river".

[0096] In addition, the present application does not limit the method for obtaining the first description text above. For example, the first description text can be pre-configured by relevant personnel for the first object description image above, so that the first description text can be directly used when the first object description image is used to train the model later.

[0097] The first posture image is used to describe the state of the object in the first object description image above (for example, which body parts it has, what postures each body part is in, and what accessories are worn on one or more body parts, etc.); and the first posture image is determined based on the first object description image above, so that the first posture image can better represent the state of the object in the first object description image.

[0098] In addition, the present application does not limit the process of determining the first posture image above. For example, it can be implemented by using any existing or future method that can determine image data that can characterize the state of an object from an image data (for example, a contour extraction method, a heat map, etc.).

[0099] In fact, in order to better improve the representation effect of the object state (for example, the degree of refinement), the present application also provides a possible implementation of the above-mentioned first posture image determination process, which may specifically include the following steps 11 and 12.

[0100] Step 11: Determine first posture description information from the first object description image.

[0101] The first posture description information is used to describe the state of the object in the first object description image (for example, which body parts it has, what posture each body part is in, and what kind of accessories one or more body parts are wearing, etc.).

[0102] In addition, the present application does not limit the implementation method of the above first posture description information, for example, it may include at least one of image acquisition information and object state representation information.

[0103] Image acquisition information refers to some parameter information (such as camera focus, camera distance and other parameter information) used when using an image acquisition device to acquire the image described by the first object above; and the present application does not limit the image acquisition information. For example, in some application scenarios, the image acquisition information may at least include image acquisition distance and / or image acquisition focus range. Among them, the image acquisition distance refers to the distance between the image acquisition device and the object when the image acquisition device is used to acquire an image of an object; and the present application does not limit the implementation method of the image acquisition distance. For example, in some application scenarios, when a certain model (such as the three-dimensional parameterized model or three-dimensional representation model below) is used to represent an object in a three-dimensional space, the image acquisition distance may refer to the distance between the image acquisition device and the model when the image acquisition device is used to acquire an image of the model. For example, in some application scenarios, when the image acquisition device is a camera, the image acquisition distance may be the camera distance. The image acquisition focus range is used to indicate which parts of an object the image acquisition device focuses on when the image acquisition device is used to acquire an image of the object; and the present application does not limit the implementation method of the image acquisition focus range. For example, when the image acquisition device is a camera, the image acquisition focus range can be the focus of the camera, so that subsequent adjustment processing of the camera focus can ensure that the camera can acquire images of the area that needs to be focused (for example, the entire object or a part of the object), so that the acquired image can describe the object state presented in the area as finely as possible (for example, the degree of finger bending, accessories worn on fingers, etc.) while ensuring the integrity of the area.

[0104] It should be noted that the present application does not limit the implementation method of the above image acquisition device. For example, in some application scenarios, the image acquisition device can be implemented by any existing or future device (such as a camera) capable of image acquisition (such as image collection). For another example, in other application scenarios, the image acquisition device can be implemented by a virtual camera (such as a virtual camera in a renderer).

[0105] The object state characterization information is used to describe the state of the first object presented in the first object description image above; and the present application does not limit the object state characterization information. For example, when the first object description image is used to describe at least one part of the first object (for example, the head, legs, etc.), the object state characterization information may at least include state description information of each part. Among them, the state description information of the nth part is used to describe the state of the nth part of the first object presented in the first object description image (for example, the degree of bending, the state of the accessories worn, etc.); and the present application does not limit the implementation method of the state description information of the nth part. n is a positive integer, n≤N, N is a positive integer, and N represents the number of parts in the "at least one part".

[0106] In addition, the present application does not limit the implementation method of the above step 11. For example, in some application scenarios, the step 11 can be specifically as follows: after obtaining the first object description image, the posture description information corresponding to the first object description image can be searched from the pre-constructed mapping relationship as the first posture description information. The mapping relationship is used to record the posture description information corresponding to different image data; and the present application does not limit the acquisition method of the posture description information corresponding to the image data. For example, it can be automatically generated, or it can be manually annotated by relevant personnel.

[0107] For example, in some other application scenarios, the above step 11 can be specifically: after acquiring the first object description image, directly use the preset posture analysis method to analyze and process the first object description image to obtain the first posture description information. The preset posture analysis method is used to analyze relevant posture information (for example, image acquisition information and / or object state representation information) from an image data; and the present application does not limit the implementation method of the preset posture analysis method, for example, it can be implemented by any existing or future method that can analyze relevant posture information from an image data.

[0108] Based on the relevant content of step 11 above, it can be known that after obtaining the first object description image, the first posture description information can be determined from the first object description image, so that the first posture description information can describe the object state in the first object description image, so that the first posture image corresponding to the first object description image can be rendered based on the first posture description information.

[0109] Step 12: Rendering the three-dimensional parameterized model using the first posture description information to obtain a first posture image, wherein the three-dimensional parameterized model is a three-dimensional digital model.

[0110] The three-dimensional parametric model is used to represent an object in three-dimensional space; and the three-dimensional parametric model belongs to a three-dimensional digital model.

[0111] In addition, the present application is not limited to the above three-dimensional parametric model. For example, when the three-dimensional parametric model is used to describe a person, the three-dimensional parametric model may be a human body model.

[0112] In addition, the present application does not limit the implementation method of the above three-dimensional parameterized model. For example, in order to better improve the three-dimensional representation effect of the object, the three-dimensional parameterized model can be implemented using a parameterized three-dimensional model such as SMPL (Skinned Multi-Person Linear). Among them, because the SMPL model not only includes the skeleton points of an object, but also includes the skin of the object, the SMPL model can better represent the state of the object in the three-dimensional space.

[0113] In addition, the present application does not limit the implementation method of the above step 12. For example, in order to better improve the posture representation effect, the step 12 can be performed with the help of dense pose (DensePose; Neverova and Kokkinos 2018) technology. Among them, DensePose technology is a pioneering technology, and the DensePose technology helps to establish a dense correspondence between a two-dimensional image and a surface-based three-dimensional model (for example, the three-dimensional parameterized model mentioned above). For example, when the three-dimensional model is an SMPL model, the DensePose technology can assign each triangular face in the SMPL model to one of the 24 predefined body parts, so that the body part corresponding to each triangular face in the SMPL model can be determined, so that the two-dimensional body image with a part label under a camera focus can be determined from some relevant areas presented in the SMPL model based on this correspondence (for example, the first posture image mentioned above). It should be noted that the present application does not limit the implementation method of the part label. For example, it can be implemented by setting the color of the corresponding part area in the two-dimensional body image so that different part areas in the two-dimensional body image have different colors.

[0114] Based on the relevant content of step 12 above, it can be known that after obtaining the first posture description information, the three-dimensional parametric model (for example, the SMPL model) can be rendered using the first posture description information to obtain a first posture image, so that the first posture image can better represent the object state described by the first posture description information.

[0115] Based on the relevant content of steps 11 to 12 above, it can be known that after obtaining the first object description image, the first posture image corresponding to the first object description image (for example, a two-dimensional body image carrying a DensePose part label) can be determined based on the first object description image, so that the first posture image can not only indicate which parts appear in the first object description image, but also indicate the state of each part (for example, what degree of bending is reached, what kind of accessories are worn, etc.), so that the first posture image can better represent the state of the object in the first object description image.

[0116] Based on the above two paragraphs, it can be known that in some application scenarios, when the above first object description image is used to describe at least one part of the first object, the first posture image corresponding to the first object description image may at least carry partial marking information corresponding to each part, and the partial marking information corresponding to different parts is different. Among them, the partial marking information corresponding to the nth part refers to the partial label marked for the nth part in the first posture image, so that the partial marking information corresponding to the nth part is used to identify the nth part; and the present application does not limit the implementation method of the partial marking information corresponding to the nth part, for example, it can be implemented by text annotation, or by adjusting the color of the area occupied by the nth part in the first posture image. n is a positive integer, n≤N, N is a positive integer, and N represents the number of parts in the "at least one part".

[0117] Based on the content of the previous paragraph, it can be seen that in one possible implementation, if the above-mentioned first object description image is used to describe at least one part of the first object, then for the first posture image corresponding to the first object description image, the colors of the areas where different parts of the first posture image are located are different, so that the purpose of adding marking information to different parts of the first posture image can be achieved, so that the first posture image can not only indicate which parts appear in the first object description image, but also indicate the state of each part, so that the first posture image can better indicate the state of the object in the first object description image.

[0118] Based on the relevant content of S101 above, it can be known that under a possible implementation mode, in order to be able to train a model with better image generation performance (for example, diffusion models such as ControlNet), it is necessary not only to obtain some image data (for example, image data in the DeepFashion dataset) and the image description text corresponding to these image data, but also to obtain the posture images corresponding to these image data (for example, two-dimensional body images carrying DensePose part labels, etc.), so that the subsequent training process for the model can be completed with the triple of (image data, image description text corresponding to the image data, posture image corresponding to the image data) as training data, so that the trained model (for example, the image generation model below) can complete the image generation processing according to the image description text + posture image, so that the trained model has better image generation performance.

[0119] S102: constructing an image generation model using a first object description image, a first description text and a first posture image, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0120] Among them, the image generation model refers to a model with image generation performance that is constructed using a triplet of (a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image) as training data.

[0121] In addition, this application does not limit the implementation method of the above image generation model. For example, it can be implemented using any existing or future diffusion model (such as ControlNet) that can perform image generation processing based on text information + additional conditions. Among them, ControlNet (Zhang and Agrawala 2023) is a diffusion model that can perform image generation processing based on text information + additional conditions.

[0122] In addition, for the above image generation model, the image generation model can be used to generate a new image data based on a binary group (an image description text, a posture image corresponding to the image description text), so that the image data not only conforms to the content described by the image description text, but also conforms to the object state described by the posture image, so that the image generation model can be used to complete the task of image generation processing based on text information + additional conditions.

[0123] It should be noted that the present application does not limit how to use the above image generation model to complete the image generation task. For example, in one possible implementation, the image generation model may include an image generation module, and the image generation module may be used to generate a new image data based on a binary group (an image description text, a posture image corresponding to the image description text), so that the image data not only conforms to the content described by the image description text, but also conforms to the object state described by the posture image, so that the image generation module can be used to complete the task of performing image generation processing based on text information + additional conditions. It can be seen that after the binary group (the above first description text, the above first posture image) is input into the image generation module, the image generation module can generate new image data based on the constraints of the first description text and the first posture image, so that the image data satisfies both the constraints described by the first description text and the constraints represented by the first posture image.

[0124] It should also be noted that the present application does not limit the implementation method of the above-mentioned image generation module. For example, in some application scenarios, the image generation module may refer to a module existing in the image generation model that can generate a new image data based on a binary group such as (an image description text, and a posture image corresponding to the image description text). For example, when the above-mentioned image generation model is a diffusion model, the image generation module may include at least one module existing in the image generation model, and the "at least one module" may be used to implement the function of generating a new image data based on a binary group such as (an image description text, and a posture image corresponding to the image description text). It should be noted that the present application does not limit the "at least one module". For example, in some application scenarios, the "at least one module" may include a denoising module.

[0125] It should be noted that the present application does not limit the method for obtaining the image generation module in the above paragraph. For example, it can be implemented by any existing or future method that can determine the image generation module from a trained diffusion model.

[0126] It should also be noted that the present application does not limit the specific implementation method of the above “generating a new image data based on the binary group (an image description text, and the posture image corresponding to the image description text)”. For example, in some application scenarios, a new image can be generated based only on the image description text and the posture image corresponding to the image description text. For another example, in other application scenarios, a new image can be generated based on a piece of noise data, an image description text, and the posture image corresponding to the image description text. For another example, in yet other application scenarios, a new image can be generated based on an image data, an image description text, and the posture image corresponding to the image description text.

[0127] In addition, the present application does not limit the construction process of the above image generation model. For example, when the image generation model is a diffusion model (such as ControlNet), the construction process of the image generation model can be implemented by using any existing or future method that can train a diffusion model based on training data. For ease of understanding, the following is an example.

[0128] As an example, when the above image generation model is a diffusion model (such as ControlNet), the construction process of the image generation model may specifically include the following steps 21 to 23.

[0129] Step 21: Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image.

[0130] It should be noted that the relevant content of step 21 can be found in S101 above, and for the sake of brevity, it will not be repeated here.

[0131] Step 22: Input the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model.

[0132] The preset diffusion model refers to a diffusion model required for use when constructing an image generation model; and the preset diffusion model is used to perform image generation processing on the input data of the preset diffusion model (e.g., the triples shown above, etc.). In addition, the present application does not limit the implementation method of the preset diffusion model, for example, it can be implemented using ControlNet.

[0133] The model output data refers to the data (e.g., generated image data and / or predicted noise, etc.) obtained by the preset diffusion model after the triplet (first object description image, first description text corresponding to the first object description image, first posture image corresponding to the first object description image) is input into the preset diffusion model above, and the preset diffusion model performs image generation processing based on the triplet. The predicted noise refers to the noise data predicted by the preset diffusion model and used in the noise addition stage of the preset diffusion model.

[0134] Step 23: Update the preset diffusion model according to the above model output data, and continue to execute the above step 21 and its subsequent steps until the first stop condition is reached, and then determine the image generation model according to the preset diffusion model.

[0135] The first stop condition refers to the condition that needs to be met when the training process for the above preset diffusion model is terminated; and the first stop condition can be preset, for example, the first stop condition may include: the model loss of the preset diffusion model is lower than the preset first loss threshold. For another example, the first stop condition may include: the rate of change of the model loss of the preset diffusion model is lower than the preset first rate of change threshold. For another example, the first stop condition may include: the number of updates of the preset diffusion model reaches the preset first number threshold.

[0136] The model loss of the preset diffusion model is used to characterize the image generation performance of the preset diffusion model; and the model loss of the preset diffusion model is usually determined based on the model output data of the preset diffusion model (e.g., generated image data and / or predicted noise, etc.). In addition, the present application does not limit the calculation method of the model loss of the preset diffusion model, which can be implemented using any existing or future loss function required for training a diffusion model.

[0137] It should be noted that the present application does not limit the implementation method of the step "updating the preset diffusion model according to the above model output data" in the above step 23. For example, it can be implemented by any existing or future diffusion model updating method. As an example, the step of "updating the preset diffusion model according to the above model output data" can be specifically: first, according to the model output data of the preset diffusion model, calculate the model loss of the preset diffusion model; then, according to the model loss, update the model parameters in the preset diffusion model, so that the preset diffusion model after the parameter update has better image generation performance.

[0138] Based on the relevant contents of steps 21 to 23 above, it can be known that in some application scenarios, the present application can use an image dataset (e.g., DeepFashion dataset), text information corresponding to each image data in the image dataset, and posture images corresponding to each image data in the image dataset (e.g., two-dimensional body images with DensePose part labels, etc.) to train ControlNet, so that the trained ControlNet has better image generation performance, for example, Figure 2 The image data generated by the trained ControlNet shown in the figure can fully demonstrate the ability of the ControlNet in generating high-quality images with consistent perspectives (for example, including various camera focuses and various body parts, such as the whole body, legs, head, etc.), so that in the subsequent case of given certain posture description information (for example, a certain camera focus, one or more body parts in a certain posture, etc.), the SMPL model with part labels can be rendered with the help of the posture description information to generate a posture image (for example, a two-dimensional body image with a DensePose part label, etc.); and then the posture image is input into the trained ControlNet as a condition, so that the trained ControlNet can generate image data that meets the condition (for example, Figure 2 2 (the image data shown).

[0139] Based on the relevant contents of S101 to S102 above, it can be known that for the image generation model construction method provided in the embodiment of the present application, first obtain a first object description image (for example, image data in a certain training image set), a first description text corresponding to the first object description image (for example, an image description text similar to "There is a stone bench by the river"), and a first posture image corresponding to the first object description image (for example, a dense posture rendering image, etc.); then use the first object description image, the first description text and the first posture image to construct an image generation model, so that the constructed image generation model can generate image data with relatively high quality based on the first description text and the first posture image, so that the image generation model can achieve the purpose of generating image data based on text information and posture information, which is beneficial to improve the image generation effect of the image generation model. Among them, because the above-mentioned first posture image is determined based on the first object description image, the first posture image can better express the state of the object in the first object description image (for example, which parts exist and what postures of each part are, etc.), so that the image data generated with the first posture image as a guiding condition is more in line with the natural posture constraints corresponding to different parts of the object (for example, the head, etc.), and thus the image data generated with the first posture image as a guiding condition has higher quality, which can effectively avoid the problems caused when generating image data only based on text information, thereby helping to improve the image generation effect.

[0140] In addition, the present application does not limit the execution subject of the above image generation model construction method. For example, any implementation of the image generation model construction method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For example, any implementation of the image generation model construction method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer, etc. The server can be a stand-alone server, a cluster server or a cloud server.

[0141] In some application scenarios, after constructing the above image generation model, the image generation model can be used to implement image processing tasks. Based on this, the present application also provides an image generation method, such as Figure 3 As shown, the image generation method may include the following S301-S303. Figure 3 A flowchart of an image generation method provided in an embodiment of the present application.

[0142] S301: Acquire a third description text and third posture description information corresponding to the third description text.

[0143] The third description text refers to the text required to be used when a new image data is generated using the constructed image generation model. It should be noted that the implementation of the third description text is similar to the implementation of the first description text above, and for the sake of brevity, it will not be repeated here.

[0144] The third posture description information refers to the posture description information that is required to be used when a new image data is generated using the constructed image generation model and has a corresponding relationship with the third description text above. It should be noted that the implementation method of the third posture description information is similar to the implementation method of the first posture description information above, and for the sake of brevity, it will not be repeated here.

[0145] S302: Rendering the three-dimensional parameterized model using the third posture description information to obtain a third posture image.

[0146] The third posture image is used to display the object state described by the third posture description information. It should be noted that the implementation of the third posture image is similar to the implementation of the first posture image, and for the sake of brevity, it will not be repeated here.

[0147] Based on the relevant content of S302 above, it can be known that for some application scenarios, after obtaining the third posture description information, the third posture description information can be used to render the above three-dimensional parametric model (for example, SMPL model) to obtain a third posture image, so that the third posture image can better represent the object state described by the third posture description information.

[0148] S303: Obtain a generated image using a pre-constructed image generation model, a third description text, and a third posture image; the image generation model is constructed using any implementation of the image generation model construction method provided in the present application.

[0149] In the present application, after obtaining the third description text and its corresponding third posture image above, the binary group (the third description text, the third posture image) can be input into a pre-constructed image generation model (for example, an image generation module in the image generation model), so that the image generation model (for example, an image generation module in the image generation model) can perform image generation processing based on the third description text and the third posture image, obtain and output a generated image, so that the generated image conforms to both the constraints described in the third description text (for example, constraints on image content) and the constraints described in the third posture image (for example, constraints on object state, for example, natural posture constraints of some body parts and / or jewelry wearing constraints, etc.), thereby effectively avoiding defects caused by the generated image data not conforming to the natural posture constraints of certain body parts (for example, generating inferior image data that does not conform to natural laws), which is conducive to improving the image generation effect.

[0150] Based on the relevant contents of S301 to S303 above, it can be known that for some application scenarios, after obtaining the constructed image generation model, the image generation model can be used to complete the task of performing image generation processing based on text + additional conditions, so that the generated image data also meets the constraints described by the text + additional conditions. Among them, because the image generation model performs image generation processing based on text + posture image, the image data generated by the image generation model not only meets the image description content defined by the text, but also meets the object state defined by the posture image (for example, the natural posture constraints of some body parts, etc.), so that the defects caused by the generated image data not meeting the natural posture constraints of some body parts (for example, generating inferior image data that does not meet the laws of nature) can be effectively avoided, which is conducive to improving the image generation effect.

[0151] In addition, the present application does not limit the execution subject of the above image generation method. For example, any implementation of the image generation method provided in the embodiments of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, any implementation of the image generation method provided in the embodiments of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0152] In fact, for the image generation model constructed above, the image generation model can not only be used to achieve image generation tasks, but also can be used to achieve three-dimensional digital model determination tasks (for example, three-dimensional digital human determination tasks). Based on this, the present application also provides a three-dimensional digital model determination method, such as Figure 4 As shown, the three-dimensional digital model determination method may include the following S401-S404. Figure 4 A flowchart of a three-dimensional digital model determination method provided in an embodiment of the present application.

[0153] S401: Rendering a three-dimensional parameterized model using second posture description information to obtain a second posture image, and rendering a three-dimensional representation model using the second posture description information to obtain a second object description image.

[0154] The second posture description information refers to the posture description information involved in the determination process of the three-dimensional digital model, so that the second posture description information can be used as the posture description information shared by the three-dimensional parameterized model and the three-dimensional representation model. It should be noted that the implementation method of the second posture description information is similar to the implementation method of the first posture description information above, and for the sake of brevity, it will not be repeated here.

[0155] In addition, the present application does not limit the method for obtaining the second posture description information mentioned above. For example, it can be set by relevant personnel according to actual application scenarios.

[0156] A 3D parametric model is a 3D digital model with fixed parameters involved in the determination of a 3D digital model (e.g. Figure 5 The three-dimensional parameterized model shown) enables the three-dimensional digital model to provide prior information (for example, Figure 5 Dense pose rendering image shown in ); and the relevant content of the three-dimensional parameterized model can be found above.

[0157] The second posture image is used to show the state of the object described by the second posture description information above; and the second posture image refers to image data obtained by rendering the three-dimensional parameterized model using the second posture description information (for example, Figure 5 It should be noted that the implementation of the second posture image is similar to the implementation of the first posture image above, and for the sake of brevity, it will not be repeated here.

[0158] It can be seen that, under a possible implementation mode, the acquisition process of the second posture image above can be specifically as follows: after obtaining the second posture description information, the three-dimensional parametric model can be rendered using the second posture description information to obtain the second posture image, so that the second posture image can better represent the object state defined by the second posture description information.

[0159] The three-dimensional representation model refers to the three-dimensional digital model involved in the determination process of the three-dimensional digital model and requiring data unit parameter update processing (for example, Figure 5The three-dimensional representation model shown in the figure); and the present application does not limit the implementation method of the three-dimensional representation model, for example, it can be implemented using a neural radiation field (Neural Radiance Fields, Nerf) grid. Wherein, the data unit refers to a data storage unit existing in the three-dimensional representation model; and the present application does not limit the implementation method of the data unit, for example, when the three-dimensional representation model is implemented using a Nerf grid, the data unit refers to a spatial grid existing in the Nerf grid.

[0160] The second object description image refers to image data required for use in the process of determining the three-dimensional digital model; and the second object description image may refer to image data obtained by rendering the three-dimensional representation model using the second posture description information (for example, Figure 5 The second object description image is used to describe at least a portion of the second object. The second object refers to the object described by the second object description image. It should be noted that the implementation of the second object is similar to the implementation of the first object above, and will not be repeated here for the sake of brevity.

[0161] It can be seen that, under a possible implementation manner, the process of acquiring the second object description image above can be specifically as follows: after acquiring the second posture description information, the three-dimensional representation model can be rendered using the second posture description information to obtain the second object description image, so that the second object description image can better describe at least one part of the second object.

[0162] It should be noted that the present application does not limit the relationship between the execution time of the acquisition process of the second posture image and the execution time of the acquisition process of the second object description image. For example, in some application scenarios, the former may be earlier than the latter. For another example, in other application scenarios, the latter may be earlier than the former. For another example, in yet other application scenarios, there is almost no difference between the two.

[0163] Based on the relevant content of S401 above, it can be known that in the process of determining the three-dimensional digital model, in order to better ensure the determination effect of the three-dimensional digital model, the three-dimensional parameterized model and the three-dimensional representation model can be rendered using the same posture description information, so that the second posture image rendered based on the three-dimensional parameterized model and the second object description image rendered based on the three-dimensional representation model can display objects under the same perspective, so that the second posture image and the second object description image can present a perspective alignment state, and then the object state described by the second posture image and the object state presented in the second object description image achieve a coherent and consistent effect, so that the second posture image can represent the object state in the second object description image, so that the second posture image and the second object description image can be used to construct a training data later, and use the training data to participate in the determination process of the three-dimensional digital model, so that the determination process of the three-dimensional digital model can be completed without the need for additional label information.

[0164] S402: Input the second object description image, the second posture image and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data based on the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image.

[0165] The second description text refers to the text information required to be used when the second object description image is used to perform image generation processing, so that the second description text can indicate what the generated image data is like. It should be noted that the implementation method of the second description text is similar to the implementation method of the first description text above, and for the sake of brevity, it will not be repeated here.

[0166] In addition, the present application does not limit the method for obtaining the second description text above. For example, it can be implemented using a text provided by relevant personnel.

[0167] The image generation model is used to perform image generation processing based on an image description text and a posture image corresponding to the image description text. For example, the image generation model can perform image generation processing based on a triple of (an image data, an image description text corresponding to the image data, and a posture image corresponding to the image data); and the image generation model is constructed using any implementation of the image generation model construction method provided in the present application. It should be noted that the relevant content of the image generation model can be found above, and for the sake of brevity, it will not be repeated here.

[0168] The model output data refers to the result (eg, generated image data and / or predicted noise) obtained by the image generation model performing image generation processing based on the triplet (second object description image, second posture image, second description text).

[0169] Based on the relevant content of S402 above, it can be known that for the current round of processing, after obtaining the second object description image, the second posture image corresponding to the second object description image, and the second description text corresponding to the second object description image, the triplet (second object description image, second posture image, second description text) can be input into a pre-constructed image generation model, so that the image generation model can perform image generation processing based on the triplet, obtain and output model output data (for example, generated image data and / or predicted noise data), so that the performance of the above three-dimensional representation model can be determined based on the model output data later.

[0170] S403: Update the three-dimensional representation model according to the model output data.

[0171] It should be noted that the present application does not limit the implementation method of the above S403. For example, when the above image generation model belongs to a diffusion model and the model output data of the image generation model includes predicted noise, the S403 may specifically include the following steps 31-32.

[0172] Step 31: Determine the SDS loss corresponding to the above three-dimensional representation model based on the predicted noise in the model output data of the above image generation model, the noise label corresponding to the above second object description image, and the Score Distillation Sampling (SDS) loss function, where the noise label refers to the noise data actually used in the process of determining the model output data.

[0173] The predicted noise refers to the noise data predicted to be used when the above image generation model performs image generation processing on the triple of (second object description image, second posture image, second description text).

[0174] The noise label refers to the noise data actually used when the above image generation model performs image generation processing on the triplet of (second object description image, second posture image, second description text).

[0175] SDS refers to a method proposed by DreamFusion (Poole et al. 2022) for assisting in generating a three-dimensional digital model based on a two-dimensional diffusion model; and the SDS loss function can be implemented using the SDS loss function shown in the following formulas (1)-(2).

[0176]

[0177]

[0178] Wherein, x=g(θ,P) represents the second object description image determined by using the three-dimensional representation model, g represents the rendering function of the three-dimensional representation model, θ represents the parameter information corresponding to each data unit in the three-dimensional representation model, P represents the second posture image mentioned above, and g(θ,P) represents the three-dimensional representation model rendered according to the second posture image; c=h(SMPL,P) represents the second posture image determined by using the three-dimensional parameterized model (for example, SMPL model), h represents the rendering function of the three-dimensional parameterized model, SMPL represents that the three-dimensional parameterized model is implemented by the SMPL model; y represents the second description text mentioned above; t represents the noise level, and z t represents the noise result obtained by the noise addition module in the image generation model for the image data x; w(t) represents the weight information determined according to t and y; ∈ φ It represents the image generation model (e.g., ControlNet) constructed above, which is used for the noise addition result z t The processing module (e.g., denoising module, etc.), φ represents the ∈ φ Parameters present in represents the predicted noise; ∈ represents the random noise data actually added when the image generation model performs noise processing on the image data x; represents the SDS loss function, is the differential symbol, is the symbol of partial derivative.

[0179] The SDS loss corresponding to the above three-dimensional characterization model is used to characterize the performance of the three-dimensional characterization model, and the SDS loss can be calculated using the SDS loss function shown in the above formulas (1)-(2).

[0180] Step 32: Update the three-dimensional representation model according to the SDS loss corresponding to the above three-dimensional representation model.

[0181] In this application, after obtaining the SDS loss corresponding to the above three-dimensional representation model, the three-dimensional representation model can be updated according to the SDS loss (for example, the parameter information corresponding to each data unit in the three-dimensional representation model is updated) so that the updated three-dimensional representation model can better represent the object state. It should be noted that the relevant content of the parameter information is shown below.

[0182] Based on the relevant content of steps 31 to 32 above, it can be known that for some application scenarios, after obtaining the model output data obtained by the image generation model for the triple (second object description image, second posture image, second description text), the SDS loss corresponding to the three-dimensional representation model can be determined based on the model output data and the SDS loss function; and then the three-dimensional representation model can be updated based on the SDS loss so that the updated three-dimensional representation model has better performance.

[0183] In fact, in some application scenarios, each data unit in the above three-dimensional representation model corresponds to some parameter information, and these parameter information can affect the object described by the three-dimensional representation model (for example, the posture of some body parts of the object, the accessories worn, the clothes worn, etc.), so in order to better improve the training effect of the three-dimensional representation model, these parameter information can be updated. Based on this, the present application also provides a possible implementation of S403 above, which may specifically include: updating the parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data of the above image generation model. Among them, for any data unit in the three-dimensional representation model, the parameter information corresponding to the data unit is used to characterize the information carried by the data unit; and the present application does not limit the parameter information corresponding to the data unit, for example, the parameter information corresponding to the data unit may include the density parameter corresponding to the data unit and the color parameter corresponding to the data unit.

[0184] Based on the content in the previous paragraph, it can be seen that for some application scenarios, after obtaining the model output data obtained by the image generation model for the triplet of (second object description image, second posture image, second description text), the SDS loss corresponding to the three-dimensional representation model can be determined based on the model output data and the SDS loss function; and then the parameter information corresponding to each data unit in the three-dimensional representation model is updated based on the SDS loss, so that the updated parameter information can better represent the object state, so that the three-dimensional representation model with these parameter information can achieve better performance.

[0185] It should be noted that, for any data unit in the three-dimensional representation model, the present application does not limit the process of determining the parameter information corresponding to the data unit. For example, when the three-dimensional representation model is implemented using a Nerf grid, the process of obtaining the parameter information corresponding to the data unit may specifically be: using a preset visualization algorithm (for example, explicitNerf and other methods) to visualize the Nerf grid and obtain at least one view stereo (for example, Figure 5The density parameters shown correspond to the view stereo and the color parameters correspond to the view stereo, etc.); then use the preset multi-layer perception algorithm to process these view stereos to obtain the density parameters and color parameters corresponding to each grid (that is, data unit) in the Nerf grid. Among them, the preset display algorithm is used to display the Nerf grid; and the present application does not limit the preset display algorithm, for example, it can be implemented using methods such as explicit Nerf or implicit nerf. The preset multi-layer perception algorithm is used to extract corresponding parameter information from these view stereos respectively; and the present application does not limit the implementation method of the preset multi-layer perception algorithm, for example, it can be implemented using a shallow multi-layer perceptron (Shallow Multilayer Perceptron, Shallow MLP).

[0186] Based on the relevant content of S403 above, it can be known that for the current round of processing, after obtaining the model output data obtained by the image generation model for performing image generation processing on the triplet of (second object description image, second posture image, second description text), the three-dimensional representation model can be updated based on the model output data so that the updated three-dimensional representation model has better performance.

[0187] S404: Determine a three-dimensional digital model according to the updated three-dimensional representation model.

[0188] In the present application, after acquiring the updated three-dimensional representation model, the updated three-dimensional representation model can be directly determined as a three-dimensional digital model, so that the three-dimensional digital model can better represent the state of the object.

[0189] Based on the relevant contents of S401 to S404 above, it can be known that for some application scenarios, after obtaining the constructed image generation model, the image generation model can perform image generation processing for any perspective (for example, a certain camera focus + some body parts in a certain posture, etc.), so that the image data at any perspective generated based on the image generation model conforms to the natural posture constraints corresponding to the perspective, so that the three-dimensional digital model determined based on the image generation model can more accurately represent the state of the object at different perspectives, thereby making the three-dimensional digital model have better performance, which is conducive to improving the determination effect of the three-dimensional digital model.

[0190] In fact, in order to better improve the determination effect of the three-dimensional digital model, the determination process for the three-dimensional digital model can be completed by means of multiple iterative updates. Based on this, the present application also provides a possible implementation of the above three-dimensional digital model determination method, which can specifically include the following steps 41 to 43.

[0191] Step 41: Rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

[0192] It should be noted that for the relevant content of step 41, please refer to the relevant content of S401 above.

[0193] Step 42: input the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model.

[0194] It should be noted that for the relevant content of step 42, please refer to the relevant content of S402 above.

[0195] Step 43: Update the three-dimensional representation model according to the model output data of the above image generation model, and return to continue to execute the image acquisition step and its subsequent steps until the second stop condition is reached, and determine the three-dimensional digital model according to the three-dimensional representation model. The image acquisition step may at least include the step of "rendering the three-dimensional representation model using the second posture description information to obtain a second object description image" in the above step 41.

[0196] Among them, the image acquisition step is used to realize the process of obtaining image data by rendering the above three-dimensional representation model (and three-dimensional parameterized model); and the image acquisition step at least includes the step of "using the second posture description information to render the three-dimensional representation model to obtain a second object description image" in the above step 41.

[0197] In addition, the present application does not limit the implementation method of the above image acquisition step. For example, in some application scenarios, the image acquisition step may refer to the above step 41. Based on this, it can be seen that in some cases, the above step 43 may specifically be: according to the model output data of the above image generation model, the three-dimensional representation model is updated, and the step 41 and its subsequent steps are returned to continue to be executed until the second stop condition is reached, and the three-dimensional digital model is determined according to the three-dimensional representation model.

[0198] In fact, the parameters in the above three-dimensional parametric model remain unchanged, so when the above second posture description information does not change, in order to better save resources (such as time, memory, computing resources, etc.), the step of "determining the second posture image from the three-dimensional parametric model rendered using the second posture description information" may not be re-executed in the new round of processing. Based on this, the present application also provides another possible implementation of the above step 43, which can be specifically: updating the three-dimensional representation model according to the model output data of the above image generation model, and returning to continue to execute the step of "rendering the three-dimensional representation model using the second posture description information to obtain the second object description image" in the above step 41 and its subsequent steps, until the second stop condition is reached, and the three-dimensional digital model is determined according to the three-dimensional representation model.

[0199] The second stop condition refers to a condition that needs to be met when the determination process of the three-dimensional digital model ends; and the second stop condition can be preset, for example, the second stop condition may include: the SDS loss corresponding to the three-dimensional digital model is lower than a preset second loss threshold. For another example, the second stop condition may include: the change rate of the SDS loss corresponding to the three-dimensional digital model is lower than a preset second change rate threshold. For another example, the second stop condition may include: the number of updates of the three-dimensional digital model reaches a preset second number threshold.

[0200] In addition, for the relevant content of the step "updating the three-dimensional representation model according to the model output data of the image generation model" in the above step 43, please refer to the relevant content of S403 above, and for the relevant content of the step "determining the three-dimensional digital model according to the three-dimensional representation model" in the above step 43, please refer to the relevant content of S404 above.

[0201] In addition, the present application does not limit the timing of determining the second stop condition above. For example, in some application scenarios, the execution time of the step of "determining whether the second stop condition is reached" may be earlier than the execution time of the step of "updating the three-dimensional representation model according to the model output data of the image generation model above" in step 43. That is, first determine whether the second stop condition is reached. If the second stop condition is not reached, update the three-dimensional representation model according to the model output data of the image generation model above, and return to continue to execute the above image acquisition step (for example, part or all of step 41) and its subsequent steps; if the second stop condition is reached, determine the three-dimensional digital model according to the three-dimensional representation model.

[0202] For another example, in some application scenarios, the execution time of the step of "determining whether the second stop condition is met" may be later than the execution time of the step of "updating the three-dimensional representation model according to the model output data of the above image generation model" in step 43. That is, after obtaining the model output data of the image generation model, the three-dimensional representation model is first updated according to the model output data, and then it is determined whether the second stop condition is met. If the second stop condition is not met, the above image acquisition step (for example, part or all of step 41) and its subsequent steps are returned to continue to be executed; if the second stop condition is met, the three-dimensional digital model is determined according to the three-dimensional representation model.

[0203] For example, in some application scenarios, the execution time of the step of "determining whether the second stop condition is reached" is almost close to the execution time of the step of "updating the three-dimensional representation model according to the model output data of the above image generation model" in step 43.

[0204] Based on the relevant contents of steps 41 to 43 above, it can be known that in some application scenarios, a pre-built image generation model can be used to assist in completing multiple rounds of iterative update processes for the three-dimensional representation model, so that the three-dimensional digital model can be determined based on the final three-dimensional representation model in the future, so that the three-dimensional digital model can better display the object, which is conducive to improving the determination effect of the three-dimensional digital model.

[0205] In fact, in order to better improve the determination effect of the three-dimensional digital model, it is necessary not only to update the parameter information corresponding to each data unit in the above three-dimensional representation model, but also to update some other information (for example, the resolution of the three-dimensional representation model, the above second posture description information, etc.), which is conducive to further improving the refinement of the three-dimensional digital model.

[0206] Based on the above content, it can be seen that the present application also provides a possible implementation of the above three-dimensional digital model determination method. Under this implementation, the three-dimensional digital model determination method may at least include the following steps 51 to 53.

[0207] Step 51: determining a second posture image from a three-dimensional parameterized model rendered using second posture description information, and determining a second object description image from a three-dimensional representation model rendered using the second posture description information.

[0208] It should be noted that for the relevant content of step 51, please refer to the relevant content of S401 above.

[0209] Step 52: input the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model.

[0210] It should be noted that for the relevant content of step 52, please refer to the relevant content of S402 above.

[0211] Step 53: Execute at least one information update operation, and return to continue to execute the image acquisition step and its subsequent steps until the second stop condition is reached, and determine the three-dimensional digital model according to the three-dimensional representation model. The image acquisition step may at least include the step of "rendering the three-dimensional representation model using the second posture description information to obtain a second object description image" in step 51 above.

[0212] For the relevant contents of the image acquisition step, please refer to step 43 above, which will not be repeated here for the sake of brevity.

[0213] The information update operation is used to implement the update process of certain information (for example, parameter information of each data unit in the three-dimensional representation model, the resolution of the three-dimensional representation model, some parameters involved in the image acquisition device, the model expression of the three-dimensional representation model, etc.).

[0214] In addition, the present application does not limit the above "at least one information update operation". For example, in some application scenarios, the "at least one information update operation" may at least include a data unit parameter update operation. The data unit parameter update operation is used to implement the update process of the parameter information corresponding to each data unit in the three-dimensional representation model. It can be seen that in one possible implementation, the data unit parameter update operation is used to implement the above step of "updating the parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data of the above image generation model".

[0215] In fact, in order to better determine a more refined three-dimensional digital model, the present application also provides an implementation of the above "at least one information update operation", under which the "at least one information update operation" not only includes the above data unit parameter update operation, but also may include one or more of a resolution improvement operation, a data unit filtering operation, a distance shortening operation, a focus range update operation, and a model expression conversion operation. These operations are described below.

[0216] The resolution enhancement operation is used to implement a resolution increase process for the three-dimensional representation model, so that the resolution enhancement operation can be used to gradually improve the resolution of the three-dimensional representation model as the three-dimensional representation model is iteratively updated. Based on this, the present application also provides a possible implementation of the update process in step 53 above, which may include at least step 531 below.

[0217] Step 531: If the preset resolution update condition is met, the resolution of the 3D representation model is updated so that the resolution of the updated 3D representation model is higher than the resolution of the 3D representation model before the update.

[0218] Among them, the preset resolution update condition refers to the condition that needs to be met when the resolution of the three-dimensional representation model needs to be increased; and the preset resolution update condition can be pre-set. For example, in some application scenarios, the preset resolution update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the third loss threshold. For another example, in some application scenarios, the preset resolution update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the third number threshold. Based on this, it can be seen that between two adjacent resolution increase processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the third number threshold.

[0219] That is, when the third numerical threshold value above is F, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Fth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the resolution of the three-dimensional representation model needs to be increased; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Fth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the resolution of the three-dimensional representation model needs to be increased; ... (and so on). Wherein, F is a positive integer, and F can be set according to the actual application scenario.

[0220] It should be noted that the present application does not limit the implementation method of increasing the resolution of the three-dimensional representation model. For example, the increase process can be implemented using the following formula (3).

[0221] The resolution of the updated three-dimensional representation model = the resolution of the three-dimensional representation model before the update + the preset resolution step (3)

[0222] The preset resolution step size refers to a single increase in resolution that is set in advance based on an application scenario.

[0223] In fact, in some application scenarios, in order to avoid too high a resolution, an upper limit of the resolution can be set. Based on this, the present application also provides two possible implementations of the above preset resolution update conditions. In one possible implementation, the preset resolution update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches a third loss threshold, and the resolution of the three-dimensional representation model does not exceed the preset resolution threshold. In another possible implementation, the preset resolution update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the third number threshold, and the resolution of the three-dimensional representation model does not exceed the preset resolution threshold.

[0224] Based on the relevant content of step 531 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model can be updated, but also the resolution of the three-dimensional representation model can be increased. In this way, the resolution of the three-dimensional representation model can be gradually improved with the iterative update process of the three-dimensional representation model. This gradual change process can be achieved, so that the three-dimensional representation model can increasingly clearly represent the object state, which is conducive to improving the determination effect of the three-dimensional digital model.

[0225] Based on the relevant content of the resolution enhancement operation above, it can be known that, in one possible implementation mode, the resolution enhancement operation is used to implement the above step of "if the preset resolution update condition is met, then the resolution of the three-dimensional representation model is updated so that the resolution of the updated three-dimensional representation model is higher than the resolution of the three-dimensional representation model before the update", so that the resolution enhancement operation can be used to complete the gradual improvement of the model resolution in the future.

[0226] The data unit filtering operation is used to implement the data unit filtering process for the three-dimensional representation model, so that the data unit filtering operation can be used to complete the gradual change process in which the number of data units of the three-dimensional representation model gradually decreases with the iterative update process of the three-dimensional representation model, which is conducive to improving the iterative update efficiency of the three-dimensional representation model. Based on this, the present application also provides a possible implementation method of the update process in step 53 above. Under this implementation method, when the parameter information corresponding to each data unit in the three-dimensional representation model includes at least density information, step 53 can at least include the following step 532.

[0227] Step 532: Search and delete a target data unit from the three-dimensional representation model, wherein a density parameter corresponding to the target data unit is lower than a preset density threshold.

[0228] The target data unit refers to a data unit existing in the above three-dimensional representation model and having a density parameter lower than a preset density threshold; and the present application does not limit the implementation method of the target data unit.

[0229] The preset density threshold refers to the threshold required for screening the worthless data units existing in the above three-dimensional representation model; and the present application does not limit the implementation method of the preset density threshold, for example, the preset density threshold may refer to a fixed value set in advance according to the application scenario. For another example, in some application scenarios, the preset density threshold may be gradually increased according to a preset rule, and the present application does not limit the preset rule, for example, it may specifically be: when the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the preset number threshold, the preset density threshold is updated so that the updated preset density threshold is greater than the preset density threshold before the update (for example, the updated preset density threshold = the preset density threshold before the update + the preset density increment).

[0230] In addition, the present application does not limit the execution method of the above step 532. For example, in one possible implementation, the step 532 can be executed once in each round of processing, so that the purpose of performing a data unit filtering operation once in each round of processing can be achieved.

[0231] In fact, in order to improve efficiency, the number of executions of the data unit filtering operation can be reduced. Based on this, the present application also provides a possible implementation of the above step 532, which can be specifically: if the preset data unit filtering condition is met, then search and delete the target data unit from the three-dimensional representation model.

[0232] Among them, the preset data unit filtering condition refers to the condition that needs to be met when performing data unit filtering processing on the three-dimensional representation model; and the preset data unit filtering condition can be pre-set. For example, in some application scenarios, the preset data unit filtering condition can specifically be: the SDS loss corresponding to the three-dimensional representation model reaches the fourth loss threshold. For another example, in some application scenarios, the preset data unit filtering condition can specifically be: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fourth number threshold. Based on this, it can be seen that between two adjacent data unit filtering processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the fourth number threshold.

[0233] That is, when the fourth number threshold value above is J, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Jth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the data unit filtering process needs to be performed on the three-dimensional representation model; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Jth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the data unit filtering process needs to be performed on the three-dimensional representation model; ... (and so on). Wherein, J is a positive integer, and J can be set according to the actual application scenario.

[0234] In fact, in some application scenarios, in order to avoid deleting too many data units, a lower limit on the number of data units can be set. Based on this, the present application also provides two possible implementations of the above preset data unit filtering conditions. In one possible implementation, the preset data unit filtering condition can specifically be: the SDS loss corresponding to the three-dimensional representation model reaches the fourth loss threshold, and the number of data units in the three-dimensional representation model is not less than the preset number threshold. In another possible implementation, the preset data unit filtering condition can specifically be: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fourth number threshold, and the number of data units in the three-dimensional representation model is not less than the preset number threshold.

[0235] In addition, in order to avoid the adverse effects caused by the accidental deletion of data units, the present application also provides a possible implementation method of the above target data unit, under which the target data unit satisfies the following conditions: the density parameter corresponding to the target data unit is lower than the preset density threshold, and the target data unit is located in the edge area of ​​the above three-dimensional representation model, so that the purpose of deleting the data units that appear in the edge area of ​​the three-dimensional representation model and have a density parameter lower than the preset density threshold can be achieved with the help of this condition, so that the problems caused by the accidental deletion of data units in the non-edge area of ​​the three-dimensional representation model can be effectively avoided, thereby helping to improve the training effect of the three-dimensional representation model. It should be noted that the present application does not limit the method for determining the edge area. For example, it can determine its edge area from the three-dimensional representation model according to a pre-set area division rule.

[0236] Based on the relevant content of step 532 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only can the parameter information corresponding to each data unit in the three-dimensional representation model be updated, but the data unit filtering process can also be performed on the three-dimensional representation model to delete the data units below the preset density threshold that appear in the edge area of ​​the three-dimensional representation model, so that the three-dimensional representation model after filtering can present the effect of the object area being tightly surrounded by a bounding box (bbox), so that the worthless data units in the three-dimensional representation model (that is, the data units that are not used to represent the object state) are as small as possible, and then the three-dimensional representation model after filtering can display the object with as few data units as possible, so that the number of data units of the three-dimensional representation model gradually decreases with the iterative update process of the three-dimensional representation model. This can effectively reduce the number of data units of the three-dimensional representation model, which is beneficial to improve the determination speed of the three-dimensional digital model.

[0237] It should be noted that the present application does not limit the implementation method of the above data unit filtering process. For example, it can adopt any method that can filter out data units whose density parameters are lower than a preset density threshold from the three-dimensional representation model (for example, Figure 6 The bounding box clipping method shown in FIG.

[0238] Based on the relevant content of the data unit filtering operation above, it can be known that under a possible implementation mode, the data unit filtering operation can be used to implement the above step of "if the preset data unit filtering conditions are met, then find and delete the target data unit from the three-dimensional representation model, and the density parameter corresponding to the target data unit is lower than the preset density threshold", so that the data unit filtering operation can be used to complete the gradual reduction of the number of model data units.

[0239] The distance shortening operation is used to achieve the process of shortening the distance between the image acquisition device and the three-dimensional representation model, so that the image acquisition device can gradually approach the three-dimensional representation model with the help of the distance shortening operation. Based on this, the present application also provides a possible implementation of the update process in step 53 above. Under this implementation, when the second posture description information above includes the image acquisition distance, step 53 can at least include the following step 533.

[0240] Step 533: If the preset distance update condition is met, the image acquisition distance in the second posture description information is updated so that the updated image acquisition distance in the second posture description information is smaller than the image acquisition distance before the update.

[0241] Among them, the preset distance update condition refers to the condition that needs to be met when the distance between the image acquisition device and the three-dimensional representation model needs to be shortened; and the preset distance update condition can be pre-set. For example, in some application scenarios, the preset distance update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the fifth loss threshold. For another example, in some application scenarios, the preset distance update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the first quantity threshold. For another example, in some application scenarios, the preset distance update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fifth number threshold. Based on this, it can be seen that between two adjacent distance shortening processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the fifth number threshold.

[0242] That is, when the fifth threshold value above is K, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Kth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the distance between the three-dimensional representation model and the image acquisition device needs to be shortened; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Kth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the distance between the three-dimensional representation model and the image acquisition device needs to be shortened; ... (and so on). Wherein, K is a positive integer, and K can be set according to the actual application scenario.

[0243] The image acquisition distance is used to represent the distance between an image acquisition device (eg, a camera, etc.) and a three-dimensional representation model.

[0244] It should be noted that the present application does not limit the implementation method of shortening the distance between the three-dimensional representation model and the image acquisition device. For example, the increase process can be implemented using the following formula (4).

[0245] Image acquisition distance after update = image acquisition distance before update - preset distance step (4)

[0246] The preset distance step refers to a single reduction amount set in advance for the image acquisition distance based on the application scenario.

[0247] In fact, in some application scenarios, in order to avoid the distance being too close, a lower limit of the distance can be set. Based on this, the present application also provides three possible implementations of the above preset distance update condition. In one possible implementation, the preset distance update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the fifth loss threshold, and the above image acquisition distance is not less than the preset distance threshold. In another possible implementation, the preset distance update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the first number threshold, and the image acquisition distance is not less than the preset distance threshold. In yet another possible implementation, the preset distance update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the fifth number threshold, and the image acquisition distance is not less than the preset distance threshold.

[0248] Based on the relevant content of step 533 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model can be updated, but also the distance between the three-dimensional representation model and the image acquisition device can be shortened. In this way, it is possible to achieve a gradual change process in which the image acquisition device gradually approaches the three-dimensional representation model during the iterative update process of the three-dimensional representation model, so that the three-dimensional representation model can better learn the detailed information related to the object, so that the three-dimensional representation model can represent the object state more and more finely, which is conducive to improving the determination effect of the three-dimensional digital model.

[0249] It should be noted that, for the above step 53, if the step 53 includes the above step 533, then because the step 533 is used to update the image acquisition distance in the above second posture description information, so that the second posture description information changes after the execution of the step 533, in order to ensure the model determination effect, it is necessary not only to re-acquire the second object description image, but also to re-acquire the second posture image, so that the "return to continue executing the image acquisition step and its subsequent steps" in the step 53 can be specifically: return to continue executing the above step 51 and its subsequent steps.

[0250] Based on the relevant content of the distance shortening operation above, it can be known that, in one possible implementation mode, the distance shortening operation is used to implement the above step of "if the preset distance update condition is met, then the image acquisition distance in the second posture description information above is updated so that the updated image acquisition distance is smaller than the image acquisition distance before the update", so that the gradual shortening of the image acquisition distance can be completed with the help of the distance shortening operation.

[0251] The focus range update operation is used to implement the focus adjustment processing for the image acquisition device, so that the focus range update operation can be used to complete the process of taking a close-up of certain parts (such as the head, etc.) of the object described by the three-dimensional representation model (or, three-dimensional parameterized model), which is conducive to improving the refinement of the three-dimensional digital model. Based on this, the present application also provides a possible implementation of the update process in step 53 above. Under this implementation, when the second posture description information above includes the image acquisition focus range, step 53 can at least include the following step 534.

[0252] Step 534: if the preset focus update condition is met, then the image acquisition focus range in the second posture description information is updated so that the updated image acquisition focus range in the second posture description information is different from the image acquisition focus range before the update.

[0253] Among them, the preset focus update condition refers to the condition that needs to be met when the focus adjustment processing is performed on the image acquisition device; and the preset focus update condition can be pre-set. For example, in some application scenarios, the preset focus update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the sixth loss threshold. For another example, in some application scenarios, the preset focus update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the second quantity threshold. For another example, in some application scenarios, the preset focus update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the sixth number threshold. Based on this, it can be seen that between two adjacent focus adjustment processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the sixth number threshold.

[0254] That is, when the sixth threshold value above is Q, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Qth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the focus adjustment process needs to be performed on the image acquisition device; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Qth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the focus adjustment process needs to be performed on the image acquisition device; ... (and so on). Wherein, Q is a positive integer, and Q can be set according to the actual application scenario.

[0255] In fact, in some application scenarios, in order to avoid the focusing range being too small, a lower limit of the focusing range can be set. Based on this, the present application also provides three possible implementations of the above preset focus update conditions. In one possible implementation, the preset focus update condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the sixth loss threshold, and the size of the focus range acquired by the above image is not less than the preset range size threshold. In another possible implementation, the preset focus update condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the second number threshold, and the size of the focus range acquired by the image is not less than the preset range size threshold. In yet another possible implementation, the preset focus update condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the sixth number threshold, and the size of the focus range acquired by the image is not less than the preset range size threshold.

[0256] In addition, the present application does not limit the implementation method of the above focus adjustment processing. For ease of understanding, two examples are used for explanation below.

[0257] Example 1, in one possible implementation, the focus adjustment process described above can be used to adjust from the global range of the object to the local range of the object. Based on this, when the second posture description information described above refers to the information required to obtain an image of the second object, the image acquisition focus range before the update is used to describe the second object as a whole, and the image acquisition focus range after the update is used to describe the first part of the second object. The first part refers to any part of the second object that needs to be focused on (for example, the head, legs, etc.).

[0258] It can be seen that in some possible implementations, when the second posture description information mentioned above refers to the information required to be based on when acquiring an image of the second object, the image acquisition focus range before the update mentioned above can be used to focus on the global range of the second object for image acquisition processing, and the image acquisition focus range after the update mentioned above can be used to focus on the first local range of the second object for image acquisition processing. Among them, the image acquisition device with the global range of the second object as the image acquisition focus range is used to perform full-view image acquisition processing on the second object. The image acquisition device with the first local range of the second object as the image acquisition focus range is used to perform close-up image acquisition processing on the first part of the second object.

[0259] Example 2, in another possible implementation, the focus adjustment process described above can be used to achieve adjustment from a local range of a certain object to a local range of another object. Based on this, it can be seen that when the second posture description information described above refers to the information required to obtain an image of the second object, the image acquisition focus range before the update is used to describe the second part of the second object, and the image acquisition focus range after the update is used to describe the third part of the second object, and the second part is different from the third part. Among them, the second part can refer to any part of the second object that needs to be focused on (for example, the head, legs, etc.); the third part can refer to any part of the second object that needs to be focused on but is different from the second part.

[0260] It can be seen that in some possible implementations, when the second posture description information mentioned above refers to the information required to be used as a basis for image acquisition of the second object, the image acquisition focus range before the update mentioned above can be used to focus on the second local range of the second object for image acquisition processing, and the updated image acquisition focus range can be used to focus on the third local range of the second object for image acquisition processing, and the third local range is different from the second local range. Among them, the image acquisition device using the second local range of the second object as the image acquisition focus range is used to perform close-up image acquisition processing on the second part of the second object. The image acquisition device using the second local range of the second object as the image acquisition focus range is used to perform close-up image acquisition processing on the third part of the second object.

[0261] In addition, the present application does not limit the determination process of the image acquisition focus range in the upper section. For example, when the image acquisition focus range in the second posture description information above is used to describe the part to be focused (for example, the head or the legs, etc.), the determination process of the image acquisition focus range may include: first determining the position information of the part to be focused from the three-dimensional parametric model above, so that the position information can indicate the position of the part to be focused in the three-dimensional parametric model; and then determining the image acquisition focus range based on the position information, so that the image acquisition device can perform complete and clear image acquisition processing as close to the part to be focused as possible based on the image acquisition focus range, so that the image data obtained by rendering the three-dimensional parametric model using the image acquisition focus range can describe the part to be focused as finely as possible while ensuring integrity, and then the image data can indicate a close-up effect of the part to be focused in the second object, so that the image data can be used to complete the refinement learning process of the three-dimensional representation model for the part to be focused in the subsequent process, which is conducive to improving the refinement of the three-dimensional representation model, thereby improving the refinement of the three-dimensional digital model.

[0262] Based on the relevant content of step 534 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model can be updated, but also the focus adjustment processing can be performed on the image acquisition device, so that the image acquisition device can obtain some close-up image data for some parts of the second object, so that the three-dimensional representation model can subsequently use these close-up image data to complete the refinement learning process of these parts, so that the three-dimensional representation model can better describe the details of these parts, which is conducive to improving the refinement of the three-dimensional representation model, thereby helping to improve the refinement of the three-dimensional digital model.

[0263] It should be noted that, for the above step 53, if the step 53 includes the above step 534, then because the step 534 is used to update the image acquisition focus range in the above second posture description information, so that the second posture description information changes after the execution of the step 534, in order to ensure the model determination effect, it is necessary not only to re-acquire the second object description image, but also to re-acquire the second posture image, so that the "return to continue executing the image acquisition step and its subsequent steps" in the step 53 can be specifically: return to continue executing the above step 51 and its subsequent steps.

[0264] Based on the above content of the focus range update operation, it can be known that in a possible implementation mode, the focus range update operation is used to implement the above step of "if the preset focus update condition is met, the image acquisition focus range is updated so that the updated image acquisition focus range is different from the image acquisition focus range before the update", so that the focus range update operation can be used to complete the learning process of some important close-up aspects (for example, Figure 6 focus mode shown).

[0265] The model expression conversion operation is used to implement the model representation conversion process for the three-dimensional representation model, so that the model expression conversion operation can be used to complete the multi-stage update processing (for example, rough stage + fine stage) of the three-dimensional representation model with gradually improved precision. Based on this, the present application also provides a possible implementation of the update process in step 53 above, which can at least include the following step 535.

[0266] Step 535: If the preset model conversion condition is met, the three-dimensional representation model is converted from the first model representation mode to the second model representation mode.

[0267] Among them, the preset model conversion condition refers to the condition that needs to be met when the model representation conversion process is performed on the three-dimensional representation model; and the preset model conversion condition can be pre-set. For example, in some application scenarios, the preset model conversion condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the seventh loss threshold. For another example, in some application scenarios, the preset model conversion condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the third quantity threshold. For another example, in some application scenarios, the preset model conversion condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the seventh number threshold. Based on this, it can be seen that between two adjacent model representation conversion processes, the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model reaches the seventh number threshold.

[0268] That is, when the seventh threshold value above is B, for a certain round of processing, if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the Bth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the model representation conversion process needs to be performed for the three-dimensional representation model; if this round of processing is used to update the parameter information corresponding to each data unit in the three-dimensional representation model for the 2Bth time, then in this round of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model needs to be updated, but also the model representation conversion process needs to be performed for the three-dimensional representation model; ... (and so on). Wherein, B is a positive integer, and B can be set according to the actual application scenario.

[0269] In fact, in some application scenarios, in order to avoid too many conversions, an upper limit on the number of conversions can be set. Based on this, the present application also provides three possible implementations of the above preset model conversion conditions. In one possible implementation, the preset model conversion condition can be specifically: the SDS loss corresponding to the three-dimensional representation model reaches the seventh loss threshold, and the number of conversions of the model expression of the three-dimensional representation model is lower than the eighth number threshold. In another possible implementation, the preset model conversion condition can be specifically: the total amount of deletions of data units in the three-dimensional representation model is an integer multiple of the third number threshold, and the number of conversions of the model expression of the three-dimensional representation model is lower than the eighth number threshold. In yet another possible implementation, the preset model conversion condition can be specifically: the number of updates of the parameter information corresponding to each data unit in the three-dimensional representation model is an integer multiple of the seventh number threshold, and the number of conversions of the model expression of the three-dimensional representation model is lower than the eighth number threshold.

[0270] The first model representation refers to the model expression of the three-dimensional representation model before conversion; and the present application does not limit the first model representation, for example, it can be a Nerf grid expression.

[0271] The second model representation refers to the model representation of the converted three-dimensional representation model; and the second model representation has a higher degree of refinement than the first model representation. In addition, the present application does not limit the second model representation, for example, it can be a triangular mesh representation.

[0272] It should be noted that the present application does not limit the implementation method of the above model representation conversion processing. For example, when the first model representation is a Nerf mesh representation and the second model representation is a triangular mesh representation, the model representation conversion processing can be specifically: first convert the Nerf mesh into a deformable tetrahedral mesh (Lin et al. 2022; Shen et al. 2021); and then determine the triangular mesh based on the deformable tetrahedral mesh.

[0273] Based on the relevant content of step 535 above, it can be known that in some application scenarios, for the information update involved in certain rounds of processing, not only the parameter information corresponding to each data unit in the three-dimensional representation model can be updated, but also the model representation conversion processing can be performed on the three-dimensional representation model. In this way, a multi-stage update processing (for example, rough stage + fine stage) with gradually increasing refinement of the three-dimensional representation model can be achieved. This is a gradual change process, which is conducive to improving the determination effect of the three-dimensional digital model.

[0274] In addition, the present application does not limit the timing of determining the second stop condition in step 53 above. For example, the implementation method of determining the second stop condition in step 53 is similar to the implementation method of determining the second stop condition in step 43 above. For the sake of brevity, it will not be repeated here.

[0275] Based on the relevant contents of steps 51 to 53 above, it can be known that in some application scenarios, if multiple rounds of iterative update processes are required for the three-dimensional representation model, then for a certain round of processing in the multiple rounds of iterative update process, it may be necessary to update at least one information in this round of processing (for example, parameter information corresponding to each data unit in the three-dimensional representation model, the resolution of the three-dimensional representation model, the distribution of data units in the three-dimensional representation model, the distance between the three-dimensional representation model and the image acquisition device, the focus of the image acquisition device, and one or more of the model representation methods of the three-dimensional representation model). In this way, an iterative update process from coarse to fine can be implemented for the three-dimensional representation model, which is conducive to determining a three-dimensional digital model with higher quality, so that the three-dimensional digital model can display the object as finely as possible under the premise of stability.

[0276] Based on the relevant content of the three-dimensional digital model determination process above, it can be seen that the present application provides a three-dimensional digital model determination method based on text and posture guidance, so that the three-dimensional digital model determination method can be used to create a high-quality and stable three-dimensional digital model, avoiding the Janus Problem that often occurs in generating three-dimensional digital models based on two-dimensional diffusion models.

[0277] In addition, the three-dimensional digital model determination method provided in the present application is implemented by using density attitude conditioning and SDS as a guide, so that the three-dimensional digital model determined by using the three-dimensional digital model determination method has higher quality.

[0278] In addition, the 3D digital model determination method provided by the present application helps to achieve high-resolution full / partial 3D digital model determination processing without the need for additional description information. At the same time, it can also easily realize animation production using the parameters of the SMPL model without further training.

[0279] In addition, the three-dimensional digital model determination method provided by the present application uses some progressive information update methods (for example, Figure 6 Some progressive information updating methods shown in the figure, etc.) improve the quality of the final three-dimensional digital model, which can ensure the model refinement process from coarse to fine, so that the final three-dimensional digital model can present finer detail information (for example, the bending range of fingers, some body accessories, etc.).

[0280] It should be noted that the progressive information updating method shown in the previous paragraph may include the following (1)-(3).

[0281] (1) The specific contents related to the progressive grid may include the following ①-③.

[0282] ①The resolution of the Nerf grid gradually increases with the iterative update process of the three-dimensional representation model.

[0283] ② In the iterative update process of the 3D representation model, the density threshold is repeatedly used to filter the scene, and the bounding box (bbox) is used to tightly surround the area. This is used to reduce the number of grids used and increase the generation speed.

[0284] ③ Progressive camera distance: The camera distance gradually decreases with the iterative update process of the 3D representation model.

[0285] (2) Close-up camera related content may include the following ④-⑥.

[0286] ④ In order to generate better details on certain parts of the body that need to be focused on, this application introduces focus in both the rough stage and the fine stage. The rough stage refers to a multi-round iterative update process with the Nerf mesh as the update object; the fine stage refers to a multi-round iterative update process with the triangular patch mesh as the update object.

[0287] ⑤Due to the SMPL prior, this application can first calculate the position of the original body part under a certain posture description information; and then move the camera close to the body part to perform loss calculation with high resolution in a very small focus area.

[0288] ⑥ Since the ControlNet built based on DensePose provided by this application has high stability, body parts can be generated without using additional computing resources, so that this application can promote the detailed learning of high-quality three-dimensional digital models with the help of focus mode (close-up camera position).

[0289] (3) The relevant contents of triangular patch mesh refinement may include the following ⑦-⑧.

[0290] ⑦ Compared with Nerf mesh, triangular mesh is used more frequently in application scenarios, so the present application can convert the Nerf mesh determined in the rough stage into a triangular mesh, and continue the iterative update processing in the fine stage based on the triangular mesh.

[0291] ⑧Integrate deformable tetrahedral meshes (Lin et al. 2022; Shen et al. 2021) to learn a textured three-dimensional representation model; and this application uses the trained explicit Nerf mesh as the initialization of the three-dimensional representation model, and optimizes the three-dimensional representation model by backpropagation using DensePose conditional SDS gradients.

[0292] Furthermore, the present application does not limit the execution subject of the above three-dimensional digital model determination method. For example, any implementation of the three-dimensional digital model determination method provided in the embodiments of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, any implementation of the three-dimensional digital model determination method provided in the embodiments of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0293] In fact, in some application scenarios (such as virtual try-on scenarios), the determined three-dimensional digital model can be rendered and displayed according to certain posture description information to meet the user's use requirements for the three-dimensional digital model in the application scenario. Based on this, the present application also provides a three-dimensional digital model display method, such as Figure 7 As shown, the method may include the following S701-S703. Figure 7 A flowchart of a three-dimensional digital model display method provided in an embodiment of the present application.

[0294] S701: Acquire fourth posture description information.

[0295] The fourth posture description information refers to the posture description information required to be provided when using a predetermined three-dimensional digital model. It should be noted that the implementation of the fourth posture description information is similar to the implementation of the first posture description information above, and for the sake of brevity, it will not be repeated here.

[0296] In addition, the present application does not limit the method for obtaining the fourth posture description information, for example, it may specifically be: the user of the terminal device inputs the fourth posture description information via an input device in the terminal device. It should be noted that the present application does not limit the input method of the fourth posture description information.

[0297] S702: Rendering a predetermined three-dimensional digital model using the fourth posture description information, where the three-dimensional digital model is determined using any implementation of the three-dimensional digital model determination method provided in the present application.

[0298] In the present application, after obtaining the fourth posture description information, the fourth posture description information can be used to render a predetermined three-dimensional digital model to obtain a rendered three-dimensional digital model, so that the rendered three-dimensional digital model can display the object according to the object state described by the fourth posture description information, thereby enabling the rendered three-dimensional digital model to better meet the user's model perspective display requirements, which is conducive to improving the user experience.

[0299] S703: Display a rendering result, where the rendering result satisfies the constraint described by the fourth posture description information.

[0300] The rendering result is obtained by rendering the three-dimensional digital model using the fourth posture description information, so that the rendering result satisfies the constraint described by the fourth posture description information.

[0301] In addition, the present application does not limit the representation method of the above rendering result. For example, in some application scenarios, the rendering result can be represented by image data. In other application scenarios, the rendering result can be represented by a three-dimensional digital model.

[0302] Based on the relevant contents of S701 to S703 above, it can be known that after the three-dimensional digital model is pre-determined, the three-dimensional digital model can be rendered according to the fourth posture description information provided by the user, so that the three-dimensional digital model can be displayed according to the object state described by the fourth posture description information, so as to meet the user's model perspective display needs. Among them, because the three-dimensional digital model can finely display the object state under various perspectives, the three-dimensional digital model can better meet the user's model perspective display needs, which is conducive to improving user experience.

[0303] In addition, the present application does not limit the execution subject of the above three-dimensional digital model display method. For example, any implementation of the three-dimensional digital model display method provided in the embodiments of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, any implementation of the three-dimensional digital model display method provided in the embodiments of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0304] Based on the image generation model construction method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation model construction device. Figure 8 Explain and illustrate. Figure 8 This is a schematic diagram of the structure of an image generation model construction device provided in an embodiment of the present application. It should be noted that for the technical details of the image generation model construction device provided in an embodiment of the present application, please refer to the relevant content of the image generation model construction method above.

[0305] like Figure 8 As shown, the image generation model construction device 800 provided in the embodiment of the present application includes:

[0306] A first acquisition unit 801 is used to acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image;

[0307] The model building unit 802 is used to build an image generation model using the first object description image, the first description text and the first posture image, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

[0308] In one possible implementation, the process of determining the first posture image includes: determining first posture description information from the first object description image; rendering a three-dimensional parameterized model using the first posture description information to obtain a first posture image, wherein the three-dimensional parameterized model is a three-dimensional digital model.

[0309] In a possible implementation manner, the first object description image is used to describe at least one part of the first object; the first posture image carries part labeling information corresponding to each part, and the part labeling information corresponding to different parts is different.

[0310] In a possible implementation manner, the colors of regions where different parts of the first posture image are located are different.

[0311] In one possible implementation, the model building unit 802 is specifically used to: input the first object description image, the first description text and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model; update the preset diffusion model according to the model output data, and continue to execute the steps of obtaining the first object description image, the first description text corresponding to the first object description image, and the first posture image corresponding to the first object description image until the first stop condition is reached, and determine the image generation model according to the preset diffusion model.

[0312] Based on the relevant content of the above-mentioned image generation model construction device 800, it can be known that for the image generation model construction device 800 provided in the embodiment of the present application, the first object description image (for example, image data in a certain training image set), the first description text corresponding to the first object description image (for example, an image description text similar to "There is a stone bench by the river"), and the first posture image corresponding to the first object description image (for example, a dense posture rendering image, etc.) are first obtained; then the first object description image, the first description text and the first posture image are used to construct an image generation model, so that the constructed image generation model can generate image data with relatively high quality based on the first description text and the first posture image, so that the image generation model can achieve the purpose of generating image data based on text information and posture information, which is beneficial to improve the image generation effect of the image generation model. Among them, because the above-mentioned first posture image is determined based on the first object description image, the first posture image can better express the state of the object in the first object description image (for example, which parts exist and what postures of each part are, etc.), so that the image data generated with the first posture image as a guiding condition is more in line with the natural posture constraints corresponding to different parts of the object (for example, the head, etc.), and thus the image data generated with the first posture image as a guiding condition has higher quality, which can effectively avoid the problems caused when generating image data only based on text information, thereby helping to improve the image generation effect.

[0313] Based on the image generation method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation device. Fig. 9 Explain and illustrate. Fig. 9 This is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. It should be noted that for the technical details of the image generating device provided in an embodiment of the present application, please refer to the relevant contents of the image generating method above.

[0314] like Fig. 9 As shown, the image generating device 900 provided in the embodiment of the present application includes:

[0315] A second acquisition unit 901 is used to acquire a third description text and third posture description information corresponding to the third description text;

[0316] A second rendering unit 902 is used to render the three-dimensional parameterized model using the third posture description information to obtain a third posture image;

[0317] The image generation unit 903 is used to obtain a generated image using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using any implementation of the image generation model construction method provided in the present application.

[0318] Based on the above-mentioned relevant contents of the image generation device 900, it can be known that for the image generation device 900 provided in the embodiment of the present application, after obtaining the constructed image generation model, the image generation model can be used to complete the task of performing image generation processing based on text + additional conditions. Among them, because the image generation model performs image generation processing based on text + posture image, so that the image data generated by using the image generation model conforms to both the image description content carried by the text and the object state described by the posture image (for example, the natural posture constraints of some body parts), it can effectively avoid the defects caused by the generated image data not conforming to the natural posture constraints of some body parts (for example, generating inferior image data that does not conform to the laws of nature), which is conducive to improving the image generation effect.

[0319] Based on the three-dimensional digital model determination method provided in the embodiment of the present application, the embodiment of the present application also provides a three-dimensional digital model determination device. Fig.10 Explain and illustrate. Fig.10 This is a schematic diagram of the structure of a three-dimensional digital model determination device provided in an embodiment of the present application. It should be noted that for the technical details of the three-dimensional digital model determination device provided in an embodiment of the present application, please refer to the relevant content of the three-dimensional digital model determination method above.

[0320] like Fig.10 As shown, the three-dimensional digital model determination device 1000 provided in the embodiment of the present application includes:

[0321] The first rendering unit 1001 is used to render the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and to render the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0322] The data determination unit 1002 is used to input the second object description image, the second posture image and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data according to the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image;

[0323] An information updating unit 1003, configured to update the three-dimensional representation model according to the model output data;

[0324] The model determination unit 1004 is used to determine a three-dimensional digital model according to the three-dimensional representation model.

[0325] In one possible implementation, the image generation model is constructed using any implementation of the image generation model construction method provided in the present application.

[0326] In a possible implementation manner, the image generation model is a diffusion model, and the model output data includes prediction noise;

[0327] The information updating unit 1003 is specifically used to: determine the SDS loss corresponding to the three-dimensional representation model based on the predicted noise, the noise label corresponding to the second object description image, and the score distillation sampling SDS loss function; the noise label refers to the noise data actually used in the process of determining the model output data; and update the three-dimensional representation model based on the SDS loss.

[0328] In a possible implementation manner, the information updating unit 1003 is specifically configured to update parameter information corresponding to each data unit in the three-dimensional representation model according to the model output data.

[0329] In a possible implementation manner, the information updating unit 1003 is further used to: after updating the three-dimensional representation model according to the model output data, return to the first rendering unit 1001 to continue to execute the image acquisition step, the image acquisition step at least including the step of rendering the three-dimensional representation model using the second posture description information to obtain a second object description image;

[0330] The model determination unit 1004 is specifically configured to determine a three-dimensional digital model according to the three-dimensional representation model until a second stop condition is reached.

[0331] In a possible implementation manner, the information updating unit 1003 is further configured to: if a preset resolution update condition is met, update the resolution of the 3D representation model so that the resolution of the updated 3D representation model is higher than the resolution of the 3D representation model before the update.

[0332] In a possible implementation manner, the information updating unit 1003 is further used to: if a preset data unit filtering condition is met, search for and delete a target data unit from the three-dimensional representation model, and a density parameter corresponding to the target data unit is lower than a preset density threshold.

[0333] In a possible implementation manner, the second posture description information includes an image acquisition distance;

[0334] The information updating unit 1003 is further configured to: if a preset distance updating condition is met, update the image acquisition distance in the second posture description information so that the updated image acquisition distance is smaller than the image acquisition distance before the update;

[0335] The image acquisition step includes: rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

[0336] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0337] The information updating unit 1003 is further configured to: if a preset focus update condition is met, update the image acquisition focus range so that the updated image acquisition focus range is different from the image acquisition focus range before the update;

[0338] The image acquisition step includes: determining a second posture image from a three-dimensional parameterized model rendered using the second posture description information, and determining a second object description image from a three-dimensional representation model rendered using the second posture description information.

[0339] In one possible implementation, the second posture description information refers to information required to be used as a basis for acquiring images of the second object; the image acquisition focus range before the update is used to describe the second object as a whole, and the image acquisition focus range after the update is used to describe the first part of the second object; or, the image acquisition focus range before the update is used to describe the second part of the second object, and the image acquisition focus range after the update is used to describe the third part of the second object.

[0340] In a possible implementation manner, the information updating unit 1003 is further configured to: if a preset model conversion condition is met, convert the three-dimensional representation model from the first model representation mode to the second model representation mode.

[0341] In a possible implementation manner, the second posture description information includes an image acquisition focus range;

[0342] If the image acquisition focus range is used to describe the part to be focused, the process of determining the image acquisition focus range includes: determining the position information of the part to be focused from the three-dimensional parameterized model; determining the image acquisition focus range based on the position information, so that the image data obtained by rendering the three-dimensional parameterized model using the image acquisition focus range is used to describe the part to be focused.

[0343] Based on the relevant content of the above-mentioned three-dimensional digital model determination device 1000, it can be known that for the three-dimensional digital model determination device 1000 provided in the embodiment of the present application, after obtaining the constructed image generation model, because the image generation model can perform image generation processing for any perspective (for example, a certain camera focus + some body parts in a certain posture, etc.), so that the image data at any perspective generated based on the image generation model conforms to the natural posture constraints corresponding to the perspective, so that the three-dimensional digital model determined based on the image generation model can more accurately represent the state of the object at different perspectives, and thus the three-dimensional digital model has better performance, which is conducive to improving the determination effect of the three-dimensional digital model.

[0344] Based on the three-dimensional digital model display method provided in the embodiment of the present application, the embodiment of the present application also provides a three-dimensional digital model display device. Fig.11 Explain and illustrate. Fig.11 This is a schematic diagram of the structure of a three-dimensional digital model display device provided in an embodiment of the present application. It should be noted that for the technical details of the three-dimensional digital model display device provided in an embodiment of the present application, please refer to the relevant content of the three-dimensional digital model display method above.

[0345] like Fig.11 As shown, the three-dimensional digital model display device 1100 provided in the embodiment of the present application includes:

[0346] The third acquisition unit 1101 is used to acquire fourth posture description information;

[0347] A model adjustment unit 1102, configured to render a predetermined three-dimensional digital model using the fourth posture description information, wherein the three-dimensional digital model is determined using any implementation of the three-dimensional digital model determination method provided in the present application;

[0348] The model display unit 1103 is used to display a rendering result, where the rendering result satisfies the constraint described by the fourth posture description information.

[0349] Based on the above-mentioned relevant contents of the three-dimensional digital model display device 1100, it can be known that for the three-dimensional digital model display device 1100 provided in the embodiment of the present application, after the three-dimensional digital model is pre-determined, the three-dimensional digital model can be rendered according to the fourth posture description information provided by the user, so that the three-dimensional digital model can be displayed according to the object state described by the fourth posture description information, so as to meet the user's model perspective display needs. Among them, because the three-dimensional digital model can finely display the object state under various perspectives, the three-dimensional digital model can better meet the user's model perspective display needs, which is conducive to improving user experience.

[0350] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the image generation model construction method provided in the present application, or executes any implementation of the three-dimensional digital model determination method provided in the present application, or executes any implementation of the image generation method provided in the present application, or executes any implementation of the three-dimensional digital model display method provided in the present application.

[0351] See also Fig.12 , which shows a schematic diagram of the structure of an electronic device 1200 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0352] like Fig.12 As shown, the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 to a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are also stored. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0353] Typically, the following devices may be connected to the I / O interface 1205: an input device 1206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1208 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 1200 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0354] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1209, or installed from the storage device 1208, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0355] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0356] An embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the image generation model construction method provided in the present application, or executes any implementation of the three-dimensional digital model determination method provided in the present application, or executes any implementation of the image generation method provided in the present application, or executes any implementation of the three-dimensional digital model display method provided in the present application.

[0357] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0358] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0359] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0360] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0361] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0362] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0363] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0364] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0365] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0366] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0367] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0368] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0369] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0370] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an image generation model, characterized in that: The method comprises: Acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image; An image generation model is constructed using the first object description image, the first description text and the first posture image. The image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

2. The method according to claim 1, characterized in that The process of determining the first posture image includes: determining first posture description information from the first object description image; The three-dimensional parameterized model is rendered using the first posture description information to obtain a first posture image, and the three-dimensional parameterized model is a three-dimensional digital model.

3. The method according to claim 1, characterized in that The first object description image is used to describe at least a portion of a first object; The first posture image carries part labeling information corresponding to each part, and the part labeling information corresponding to different parts is different.

4. The method according to claim 3, characterized in that The colors of regions where different parts of the first posture image are located are different.

5. The method according to any one of claims 1 to 4, characterized in that: The construction process of the image generation model includes: Inputting the first object description image, the first description text, and the first posture image into a preset diffusion model to obtain model output data of the preset diffusion model; According to the model output data, the preset diffusion model is updated, and the steps of acquiring a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image are continued until a first stop condition is reached, and the image generation model is determined according to the preset diffusion model.

6. A method for determining a three-dimensional digital model, characterized in that: The method comprises: Rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image; Inputting the second object description image, the second posture image, and the second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is used to generate image data according to the second posture image and the second description text, and the image data satisfies the constraints described by the second description text and the constraints represented by the second posture image; updating the three-dimensional representation model according to the model output data; A three-dimensional digital model is determined according to the three-dimensional representation model.

7. The method according to claim 6, characterized in that The image generation model is constructed using the image generation model construction method described in any one of claims 1-5.

8. The method according to claim 6, characterized in that The image generation model belongs to a diffusion model, and the model output data includes prediction noise; The updating of the three-dimensional representation model according to the model output data comprises: Determining the SDS loss corresponding to the three-dimensional representation model according to the predicted noise, the noise label corresponding to the second object description image, and the score distillation sampling SDS loss function; the noise label refers to the noise data actually used in the process of determining the output data of the model; The three-dimensional representation model is updated according to the SDS loss.

9. The method according to claim 6, characterized in that The updating of the three-dimensional representation model according to the model output data comprises: According to the model output data, parameter information corresponding to each data unit in the three-dimensional representation model is updated.

10. The method according to any one of claims 6 to 9, characterized in that: After updating the three-dimensional representation model according to the model output data, the method further includes: Continue to execute the image acquisition step, the image acquisition step at least includes the step of rendering the three-dimensional representation model using the second posture description information to obtain a second object description image; Determining a three-dimensional digital model according to the three-dimensional representation model includes: When the second stop condition is reached, a three-dimensional digital model is determined according to the three-dimensional representation model.

11. The method according to claim 10, characterized in that Before continuing to perform the image acquisition step, the method further includes: If the preset resolution update condition is met, the resolution of the three-dimensional representation model is updated so that the resolution of the updated three-dimensional representation model is higher than the resolution of the three-dimensional representation model before the update.

12. The method according to claim 10, characterized in that Before continuing to perform the image acquisition step, the method further includes: Searching and deleting a target data unit from the three-dimensional representation model, wherein a density parameter corresponding to the target data unit is lower than a preset density threshold; or, If the preset data unit filtering condition is met, the target data unit is searched and deleted from the three-dimensional representation model.

13. The method according to claim 10, characterized in that The second posture description information includes an image acquisition distance; Before continuing to perform the image acquisition step, the method further includes: If the preset distance update condition is met, updating the image acquisition distance in the second posture description information so that the updated image acquisition distance is smaller than the image acquisition distance before the update; The step of continuing to perform the image acquisition step comprises: Continue to perform the steps of rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

14. The method according to claim 10, characterized in that The second posture description information includes an image acquisition focus range; Before continuing to perform the image acquisition step, the method further includes: If the preset focus update condition is met, updating the image acquisition focus range so that the updated image acquisition focus range is different from the image acquisition focus range before the update; The step of continuing to perform the image acquisition step comprises: Continue to perform the steps of rendering the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and rendering the three-dimensional representation model using the second posture description information to obtain a second object description image.

15. The method according to claim 14, characterized in that The second posture description information refers to information required to be based on when acquiring the second object; The image acquisition focus range before updating is used to describe the whole of the second object, and the image acquisition focus range after updating is used to describe the first part of the second object; or, the image acquisition focus range before updating is used to describe the second part of the second object, and the image acquisition focus range after updating is used to describe the third part of the second object.

16. The method according to claim 10, characterized in that Before continuing to perform the image acquisition step, the method further includes: If the preset model conversion condition is met, the three-dimensional representation model is converted from the first model representation mode to the second model representation mode.

17. The method according to claim 6, characterized in that The second posture description information includes an image acquisition focus range; If the image acquisition focus range is used to describe the portion to be focused, the process of determining the image acquisition focus range includes: Determining position information of the portion to be focused from the three-dimensional parameterized model; The image acquisition focus range is determined according to the position information, so that the image data obtained by rendering the three-dimensional parameterized model using the image acquisition focus range is used to describe the part to be focused.

18. An image generation method, characterized in that: The method comprises: Acquire a third description text and third posture description information corresponding to the third description text; Rendering the three-dimensional parameterized model using the third posture description information to obtain a third posture image; A generated image is obtained using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using the image generation model construction method described in any one of claims 1-5.

19. An image generation model construction device, characterized in that: include: A first acquisition unit, configured to acquire a first object description image, a first description text corresponding to the first object description image, and a first posture image corresponding to the first object description image, wherein the first posture image is determined based on the first object description image; A model building unit is used to build an image generation model using the first object description image, the first description text and the first posture image, wherein the image generation model is used to generate image data based on the first description text and the first posture image, and the image data satisfies the constraints described by the first description text and the constraints represented by the first posture image.

20. A three-dimensional digital model determination device, characterized in that: include: A first rendering unit, configured to render the three-dimensional parameterized model using the second posture description information to obtain a second posture image, and to render the three-dimensional representation model using the second posture description information to obtain a second object description image; a data determination unit, configured to input the second object description image, the second posture image, and a second description text corresponding to the second object description image into a pre-constructed image generation model to obtain model output data of the image generation model; the image generation model is configured to generate image data according to the second posture image and the second description text, the image data satisfying the constraints described by the second description text and the constraints represented by the second posture image; An information updating unit, configured to update the three-dimensional representation model according to the model output data; The model determination unit is used to determine the three-dimensional digital model according to the three-dimensional representation model.

21. An image generating device, characterized in that: include: A second acquisition unit, used to acquire a third description text and third posture description information corresponding to the third description text; A second rendering unit, configured to render the three-dimensional parameterized model using the third posture description information to obtain a third posture image; An image generation unit is used to obtain a generated image using a pre-constructed image generation model, the third description text and the third posture image; the image generation model is constructed using the image generation model construction method described in any one of claims 1-5.

22. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 18.

23. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 18.

Images