Image generation method and device, equipment, medium and product

Abstract

The invention discloses an image generation method and device, equipment, a medium and a product. The method comprises the steps of firstly obtaining a target image and state parameters of an object in the target image; according to state adjustment information specified for the object, the state parameters are adjusted, and adjusted parameters are obtained; and then, according to the adjusted parameters and the target image, obtaining a generated image. Wherein the three-dimensional model with the state parameters is used for displaying at least one candidate state of the whole object in the target image in the three-dimensional space, so that the state parameters can represent the state of the whole object in the target image, and the state of the whole object can be adjusted by adjusting the state parameters. Therefore, the generated image obtained based on the adjusted parameters can better meet the overall adjustment requirement of the object in one image, all parts of the object can be integrally coordinated and unified for uniform deformation, and the image adjustment effect can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image generation method, apparatus, device, medium, or product. Background Technology

[0002] For some application scenarios, such as image shooting, video shooting, image beautification, or image modification, there is a need to adjust an object in an image, such as making it thinner or fatter. Summary of the Invention

[0003] This application provides an image generation method, apparatus, device, medium, and product, which helps to improve the image adjustment effect.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] This application provides an image generation method, the method comprising: acquiring a target image and state parameters of an object in the target image; a three-dimensional model having the state parameters being used to display at least one candidate state of the object as a whole in the target image in three-dimensional space, the at least one candidate state including body shape and / or posture; adjusting the state parameters according to state adjustment information specified for the object to obtain adjusted parameters, the state adjustment information being used to indicate adjustment processing for a target state among the at least one candidate state, the target state described by the adjusted parameters being different from the target state described by the state parameters; generating an image based on the adjusted parameters and the target image; the target state described by the generated image being consistent with the target state described by the adjusted parameters, and other information described by the generated image other than the target state being consistent with other information described by the target image other than the target state.

[0006] In one possible implementation, the state parameters include parameters corresponding to the target state;

[0007] The step of adjusting the state parameters based on the state adjustment information specified for the object to obtain the adjusted parameters includes: adjusting the parameters corresponding to the target state based on the state adjustment information to obtain the adjustment result corresponding to the target state; and determining the adjusted parameters based on the adjustment result corresponding to the target state, wherein the adjusted parameters include the adjustment result corresponding to the target state.

[0008] In one possible implementation, the state parameters further include parameters corresponding to other states among the at least one candidate state besides the target state; the adjusted parameters further include parameters corresponding to the other states.

[0009] In one possible implementation, the target state is body shape; the state adjustment information is used to instruct adjustments to be made to all or part of the shape parameters of the body shape, except for height.

[0010] After obtaining the adjustment result corresponding to the target state, the method further includes: determining the adjusted height of the object based on the adjustment result corresponding to the target state; updating the adjustment result corresponding to the target state based on the ratio between the adjusted height of the object and the original height of the object, to obtain the updated result corresponding to the target state, wherein the height described by the updated result is consistent with the original height, and the original height is determined based on the parameters corresponding to the target state;

[0011] The step of determining the adjusted parameters based on the adjustment result corresponding to the target state includes: determining the adjusted parameters based on the update result corresponding to the target state, wherein the adjusted parameters include the update result corresponding to the target state.

[0012] In one possible implementation, the method further includes: obtaining the state feature extraction result of the adjusted parameters, the semantic information extraction result of the target image, and the image feature extraction result of the target image;

[0013] The step of obtaining a generated image based on the adjusted parameters and the target image includes: obtaining the generated image based on the state feature extraction result, the semantic information extraction result, and the image feature extraction result.

[0014] In one possible implementation, the generated image is obtained using a first network and a second network; the first network is used to process the image feature extraction result, and the semantic information extraction result is injected into the decoding module of the first network through cross-attention; the second network is used to process the state feature extraction result, and the semantic information extraction result is injected into the decoding module of the second network through cross-attention, and the output data of the decoding module of the first network is injected into the decoding module of the second network through cross-attention.

[0015] In one possible implementation, the decoding module includes a first submodule and a second submodule, the input data of the second submodule includes the output data of the first submodule; the semantic information extraction result is injected into the second submodule of the decoding module of the first network through a cross-attention method; the output data of the first submodule in the decoding module of the first network is injected into the first submodule of the decoding module of the second network through a cross-attention method; the semantic information extraction result is injected into the second submodule of the decoding module of the second network through a cross-attention method.

[0016] In one possible implementation, the second network is used to process the noisy image and the state feature extraction results.

[0017] In one possible implementation, the network structure of the first network is the same as that of the second network.

[0018] In one possible implementation, the process of determining the state feature extraction result of the adjusted parameters includes:

[0019] Image acquisition is performed on the three-dimensional model with the adjusted parameters to obtain a two-dimensional image;

[0020] The two-dimensional image is subjected to feature extraction processing to obtain the state feature extraction result of the adjusted parameters.

[0021] In one possible implementation, obtaining the generated image based on the adjusted parameters and the target image includes: processing the adjusted parameters and the target image using a target model to obtain the generated image.

[0022] In one possible implementation, the target model is used to process the noisy image, the adjusted parameters, and the target image to obtain the generated image;

[0023] The training process of the target model includes: acquiring a first image, a second image, target noise, a noise-added result of the second image, and state parameters of the second image, wherein the noise-added result is obtained by adding noise to the second image using the target noise, and the state parameters of the second image are used to describe at least one candidate state of the object as a whole in the second image; processing the noise-added result, the first image, and the state parameters of the second image using the target model to obtain predicted noise; updating the target model based on the difference between the predicted noise and the target noise, and continuing to execute the steps of acquiring the first image, the second image, the target noise, the noise-added result of the second image, and the state parameters of the second image until a preset stopping condition is reached.

[0024] In one possible implementation, the first image and the second image are extracted from the same video.

[0025] In one possible implementation, the target model includes a state feature extraction module, a semantic information extraction module, an image feature extraction module, a first network, and a second network; the state feature extraction module is used to obtain the state feature extraction result of the adjusted parameters; the semantic information extraction module is used to obtain the semantic information extraction result of the target image; the image feature extraction module is used to obtain the image feature extraction result of the target image; the input data of the first network includes the image feature extraction result and the semantic information extraction result; the input data of the second network includes the state feature extraction result, the semantic information extraction result, and the output data of some or all modules in the first network.

[0026] This application provides an image generation apparatus, comprising:

[0027] The first acquisition unit is used to acquire a target image and state parameters of objects in the target image. A three-dimensional model with the state parameters is used to display at least one candidate state of the object as a whole in the target image in three-dimensional space. The at least one candidate state includes body shape and / or posture.

[0028] A parameter adjustment unit is used to adjust the state parameters according to the state adjustment information specified for the object to obtain the adjusted parameters. The state adjustment information is used to indicate the adjustment process for the target state among the at least one candidate state. The target state described by the adjusted parameters is different from the target state described by the state parameters.

[0029] An image generation unit is configured to generate an image based on the adjusted parameters and the target image; the target state described by the generated image is consistent with the target state described by the adjusted parameters, and other information described by the generated image other than the target state is consistent with other information described by the target image other than the target state.

[0030] This application provides an electronic device, the device comprising: a processor and a memory;

[0031] The memory is used to store instructions or computer programs;

[0032] The processor is configured to execute the instructions or computer program in the memory, so that the electronic device performs the image generation method provided in this application.

[0033] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the image generation method provided in this application.

[0034] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the image generation method provided in this application.

[0035] Compared with related technologies, this application has at least the following advantages:

[0036] In the technical solution provided in this application, the target image and the state parameters of the object in the target image are first obtained; then, based on the state adjustment information specified for the object, such as the body being 0.5 times thinner, the state parameters are adjusted to obtain the adjusted parameters; then, based on the adjusted parameters and the target image, the generated image is obtained.

[0037] In this process, a 3D model with state parameters is used to display at least one candidate state of the object in the target image in 3D space, such as body shape and / or posture. This allows the state parameters to represent the overall state of the object in the target image to a certain extent. Consequently, the adjustment processing of the state parameters can adjust the overall state of the object. Furthermore, the adjusted parameters obtained based on the adjustment processing can represent the adjustment requirements for the overall state of the object. This allows the generated image obtained based on the adjusted parameters to meet the adjustment requirements, thus better satisfying the overall adjustment requirements of the object in an image. This enables the overall coordination and uniform deformation of various parts of the object, thereby improving the image adjustment effect.

[0038] Furthermore, since the state adjustment information is used to instruct adjustment processing for a target state, such as body shape, among at least one candidate state, so that the adjusted parameters obtained based on the state adjustment information can better meet the state adjustment requirements of the target state described by the state adjustment information, the target state described by the adjusted parameters is different from the target state described by the state parameters. Consequently, the target state described by the generated image based on the adjusted parameters is consistent with the target state described by the adjusted parameters, and other information described by the generated image besides the target state is consistent with other information described by the target image besides the target state. Thus, the generated image can better meet the overall adjustment requirements of an object in an image, such as the overall body shape adjustment requirements, thereby improving the image adjustment effect. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating an image generation method provided in this application embodiment;

[0041] Figure 2 A schematic diagram illustrating an adjustment implementation scheme provided in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] Research has revealed that some implementation schemes for adjusting objects in an image, such as the liquefaction scheme, can only perform localized deformations on objects, such as slimming the waist or face. When overall deformation of the object is needed, such as slimming the entire object, the user must manually adjust each part of the object individually. This results in significant abruptness between the adjustments of different parts, making it impossible to achieve a unified and symmetrical deformation of all parts of the object, thus affecting the overall image adjustment effect.

[0045] Based on the above research, in order to better improve the image adjustment effect, this application provides an image generation method, which includes: firstly acquiring a target image and the state parameters of the object in the target image; then adjusting the state parameters according to the state adjustment information specified for the object, such as the body being 0.5 times thinner, to obtain the adjusted parameters; and finally, generating an image based on the adjusted parameters and the target image.

[0046] In this process, a 3D model with state parameters is used to display at least one candidate state of the object in the target image in 3D space, such as body shape and / or posture. This allows the state parameters to represent the overall state of the object in the target image to a certain extent. Consequently, the adjustment processing of the state parameters can adjust the overall state of the object. Furthermore, the adjusted parameters obtained based on the adjustment processing can represent the adjustment requirements for the overall state of the object. This allows the generated image obtained based on the adjusted parameters to meet the adjustment requirements, thus better satisfying the overall adjustment requirements of the object in an image. This enables the overall coordination and uniform deformation of various parts of the object, thereby improving the image adjustment effect.

[0047] Furthermore, since the state adjustment information is used to instruct adjustment processing for a target state, such as body shape, among at least one candidate state, so that the adjusted parameters obtained based on the state adjustment information can better meet the state adjustment requirements of the target state described by the state adjustment information, the target state described by the adjusted parameters is different from the target state described by the state parameters. Consequently, the target state described by the generated image based on the adjusted parameters is consistent with the target state described by the adjusted parameters, and other information described by the generated image besides the target state is consistent with other information described by the target image besides the target state. Thus, the generated image can better meet the overall adjustment requirements of an object in an image, such as the overall body shape adjustment requirements, thereby improving the image adjustment effect.

[0048] Furthermore, this application does not limit the executing entity of the image generation method provided in the embodiments of this application. For example, the image generation method provided in the embodiments of this application can be applied to a terminal device or a server. Alternatively, the image generation method provided in the embodiments of this application can also be implemented through a data interaction process between a terminal device and a server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0049] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0050] To better understand the technical solution provided in this application, the image generation method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the image generation method provided in this application includes the following steps S1-S3. Wherein, the... Figure 1 This is a flowchart of an image generation method provided in an embodiment of this application.

[0051] S1: Obtain the target image and the state parameters of the objects in the target image. A 3D model with the state parameters is used to display at least one candidate state of the object as a whole in the target image in 3D space. The at least one candidate state includes body shape and / or posture.

[0052] The target image is used to describe the state of an object, such as a person, a virtual avatar, or an animal, including its size and / or posture. It should be noted that this application does not limit the implementation of the object; for example, the object can represent the foreground described by the target image, such as a person, a digital human, or an animal. Furthermore, this application does not limit the implementation of the object; for example, the object can be implemented as an animal, a virtual avatar, or an object.

[0053] As can be seen, in one possible implementation, the target image includes an object in a first state. The first state refers to the state of the object as presented in the target image, such as body shape and / or posture; moreover, this application does not limit the first state, for example, the first state may include a first body shape and a first posture. The first body shape refers to the body shape of the object as presented in the target image, such as height, weight, etc. The first posture refers to the overall posture of the object as presented in the target image.

[0054] Additionally, in status adjustment tasks triggered by a user targeting a specific image, such as the task of becoming 0.5 times thinner, the target image can refer to an image that requires status adjustment processing, such as... Figure 2 The image 1 shown is used to provide information other than the adjusted state (such as body shape), such as the unadjusted state (such as pose), background, object ID, etc., in the image generation process involved in the state adjustment task, as shown in S3 below.

[0055] It should be noted that this application does not limit the implementation of the object ID in the above paragraph. For example, the object ID is used to describe some identifying information of the object in the target image, such as appearance, clothing, and video, so that the object ID can describe the identifying characteristics of the object in the target image, such as facial features.

[0056] Furthermore, this application does not limit the implementation method of the target image. For example, in some application scenarios, such as image shooting scenarios, video shooting scenarios, image beautification scenarios, or video processing scenarios, the target image may refer to an image provided by the user through a certain input device, so that the image can be adjusted according to the user's needs, such as making the whole body thinner.

[0057] The object in the target image refers to the object described by the target image, such as a person, a digital person, or an animal. Furthermore, the target image can describe at least one candidate state of the object as a whole, such as body shape and / or posture. Here, "the object as a whole" refers to the object's body, such as all parts of the object. Therefore, in one possible implementation, the object as a whole may include at least two parts of the object, such as all parts.

[0058] At least one candidate state is used to describe the characteristics of an object, and this application does not limit the implementation of the at least one candidate state. For example, the at least one candidate state may include body shape and / or posture. Body shape describes the characteristics of the object in terms of body shape, such as height, weight, etc. Posture describes the characteristics of the various parts of the object in terms of posture, such as arms outstretched, etc.

[0059] The state parameters of an object in the target image are used to describe at least one candidate state of the object as a whole in the target image, such as body shape and / or pose, so that a 3D model having the state parameters can be used to display at least one candidate state of the object as a whole in the target image in 3D space. The at least one candidate state includes body shape and / or pose.

[0060] A three-dimensional model is used to show the state of an object in three-dimensional space; and this application does not limit the implementation of the three-dimensional model. For example, it can be implemented using any existing or future three-dimensional model that can change its state based on parameters.

[0061] For example, to better improve the overall adjustment effect, the aforementioned 3D model can be implemented using an SMPL (Skinned Multi-Person Linear) model. This allows for subsequent adjustments to some or all parameters of the 3D model to achieve overall state adjustment. SMPL is a 3D model construction technique; it can describe the state of a 3D object using two types of statistical parameters: body shape parameter β and posture parameter θ. The body shape parameter β includes data in 10 dimensions, allowing it to describe the overall shape of an object, such as its body shape. Each dimension of the body shape parameter β can be interpreted as the object's overall state under a certain shape index, such as height or weight, thus enabling the body shape parameter β to describe the object's state under multiple shape indices. The pose parameter θ includes 24×3 dimensions of data, so that the pose parameter θ can describe the overall motion pose of an object through the 24×3 dimensions of data. The 24 in "24×3" refers to 23 joints + 1 root node, and the 3 in "24×3" refers to the value in the axis-angle.

[0062] Based on the above three paragraphs, it can be seen that in one possible implementation, in order to better represent the overall state of the object, the state parameters of the object in the target image can be implemented using SMPL parameters, so that the state parameters can include body shape parameter β and posture parameter θ, thereby enabling the state parameters to represent the overall body shape and posture of the object in the target image, and thus enabling the 3D model with the state parameters to better display the overall state of the object in 3D space, such as body shape and posture.

[0063] Furthermore, this application does not limit the implementation of the state parameters mentioned above. For example, when the state parameters of an object in a target image are used to describe at least one candidate state of the object as a whole in the target image, the state parameters may include parameters corresponding to the at least one candidate state. The parameter corresponding to the i-th candidate state is used to describe the i-th candidate state of the object as a whole in the target image, such as body shape or posture, where i is a positive integer, i≤I, I is a positive integer, and I represents the number of states among the at least one candidate state.

[0064] As can be seen, in one possible implementation, the state parameters of the object in the target image mentioned above may include parameters corresponding to body shape and / or parameters corresponding to posture. The body shape parameter describes the overall body shape of the object in the target image, such as height, weight, etc. Furthermore, this application does not limit the implementation of the body shape parameter; for example, the body shape parameter can be implemented using the aforementioned body shape parameter β, so that the body shape parameter can describe the state of the object in the target image under multiple shape indicators (e.g., height, weight, etc.). The posture parameter describes the overall posture of the object in the target image, such as the posture of each part; furthermore, this application does not limit the implementation of the posture parameter; for example, the posture parameter can be implemented using the aforementioned posture parameter θ, so that the posture parameter can better represent the overall action posture of the object in the target image.

[0065] Furthermore, this application does not limit the method of obtaining the aforementioned state parameters. For example, it can specifically involve performing parameter prediction processing on the target image to obtain the state parameters of the object in the target image. This parameter prediction processing is used to predict the state parameters of the object in the target image. Moreover, this application does not limit the parameter prediction processing; for example, it can be implemented using any existing or future parameter prediction method, such as any method for predicting 3D model parameters.

[0066] Furthermore, to better improve the overall adjustment effect, the state parameters of the object in the target image mentioned above can be obtained using SMPL technology. Therefore, in one possible implementation, the process of obtaining these state parameters can be as follows: perform SMPL prediction processing on the target image to obtain the state parameters of the object in the target image, such that these state parameters include the body shape parameter β and the pose parameter θ, thereby making the 3D model with these state parameters an SMPL model, and thus enabling the 3D model with these state parameters to better represent the overall state of the object in the target image in 3D space.

[0067] It should be noted that this application does not limit the implementation method of the SMPL prediction processing in the above paragraph. For example, it can be implemented using any existing or future method that can predict the SMPL parameters of an image, such as by using a pre-built machine learning model with SMPL prediction processing function.

[0068] Based on the content related to S1 above, in some scenarios, such as image editing, upon receiving a target image from the user that needs adjustment, such as... Figure 2Following the image shown in Figure 1, SMPL prediction processing is performed on the target image to obtain the state parameters of the object in the target image. These state parameters include parameters corresponding to body shape and parameters corresponding to posture, so that the state parameters can represent the state of the object in the target image, such as body shape and posture. This allows for subsequent adjustments to the body shape and / or posture of the object in the target image based on these state parameters.

[0069] S2: Based on the state adjustment information specified for the object, the state parameters are adjusted to obtain the adjusted parameters. The state adjustment information is used to indicate the adjustment of the target state among at least one candidate state. The target state described by the adjusted parameters is different from the target state described by the state parameters.

[0070] The state adjustment information describes the state adjustment requirements specified by the user for an object in the target image, such as body shape adjustment requirements and / or posture adjustment requirements. This state adjustment information is used to instruct adjustment processing for some or all of the at least one candidate state, such as the target state. The target state refers to a state existing among the at least one candidate state that requires adjustment, such as body shape and / or posture. Furthermore, this application does not limit the implementation of the target state; for example, the target state may include body shape.

[0071] Furthermore, this application does not limit the implementation method of the above-mentioned state adjustment information. For example, it may include body shape adjustment information and / or posture adjustment information.

[0072] Body shape adjustment information is used to describe the body shape adjustment requirements specified by the user for an object in a target image, such as becoming 0.5 times thinner. Furthermore, this application does not limit the implementation method of the body shape adjustment information. For example, the body shape adjustment information may include an adjustment intensity value. This adjustment intensity value can be any value within the range [-1, 1], so that the adjustment intensity value can represent the degree to which the object becomes fatter or thinner. For example, if the adjustment intensity value is -0.5, then the adjustment intensity value can represent that the object becomes 0.5 times thinner; if the adjustment intensity value is 0.5, then the adjustment intensity value can represent that the object becomes 0.5 times fatter.

[0073] The pose adjustment information describes the pose adjustment requirements specified by the user for an object in a target image, such as selecting a candidate pose. This application does not limit the implementation method of the pose adjustment information; for example, the pose adjustment information may include a target identifier. This target identifier refers to the pose identifier selected by the user from a pre-defined set of candidate pose identifiers, so that the target identifier is used to identify the desired pose, thus enabling the selection of a pose that meets the user's requirements from a set of pre-defined candidate poses. Alternatively, the pose adjustment information may include pose description text. This pose description text is used to describe the pose adjustment requirements in textual form.

[0074] Furthermore, this application does not limit the method of obtaining the status adjustment information mentioned above. For example, the status adjustment information may be obtained based on the content entered by the user through the interactive interface, such as the adjustment intensity value of -0.5.

[0075] The adjusted parameter refers to the result obtained by adjusting the state parameter according to the state adjustment information above, so that there is a difference between the adjusted parameter and the state parameter, thereby enabling this difference to represent the state adjustment requirement described by the state adjustment information. It should be noted that this application does not limit the implementation method of the adjusted parameter; for example, the implementation method of the adjusted parameter may be similar to the implementation method of the state parameter above.

[0076] In addition, when the above state adjustment information is used to indicate the adjustment process for the target state among at least one candidate state, the above adjusted parameter must at least satisfy the following constraint: the target state described by the adjusted parameter is different from the target state described by the state parameter, so that the adjusted parameter can better represent the state adjustment requirements of the target state described by the state adjustment information.

[0077] Furthermore, this application does not limit the implementation of S2 above. For example, when the state parameters above include parameters corresponding to the target state, such as body size parameter β, if the state adjustment information above is used to indicate adjustment processing for the target state, then S2 may specifically include steps 11-12 below.

[0078] Step 11: Based on the status adjustment information above, adjust the parameters corresponding to the target status to obtain the adjustment result corresponding to the target status.

[0079] The adjustment result corresponding to the target state is obtained by adjusting the parameters corresponding to the target state according to the state adjustment information above, so that the state described by the adjustment result corresponding to the target state is different from the state described by the parameters corresponding to the target state, and so that the adjustment result corresponding to the target state can, to a certain extent, represent the state adjustment requirements of the target state described by the state adjustment information.

[0080] Furthermore, this application does not limit the implementation of step 11 above. For example, it can be implemented using any existing or future parameter adjustment method, such as a method for adjusting the SMPL parameter.

[0081] In addition, in order to better achieve overall state balance adjustment, this application also provides a possible implementation of step 11 above, in which step 11 may specifically include steps 111-112 below.

[0082] Step 111: Based on the status adjustment information above, determine the offset of the parameters corresponding to the target status.

[0083] The offset of the parameter corresponding to the target state is used to describe the difference between the parameter corresponding to the target state and the parameter expected by the state adjustment information, so that the parameter corresponding to the target state can be adjusted according to the offset to obtain the parameter expected by the state adjustment information, such as the adjustment result of the target state above.

[0084] It should be noted that, for the "parameters expected by the state adjustment information" in the above paragraph, these "parameters expected by the state adjustment information" refer to parameters that can satisfy the state adjustment requirements described by the state adjustment information, such as the adjustment results corresponding to the target state below.

[0085] Furthermore, this application does not limit the implementation method of the offset of the parameter corresponding to the target state mentioned above. For example, when the parameter corresponding to the target state includes data in multiple dimensions, the offset includes the offset of each dimension. For ease of understanding, the following explanation is provided with reference to examples.

[0086] As an example, when the target state is body shape, and the parameters corresponding to the target state include data in 10 dimensions, the offset of the parameters corresponding to the target state includes the offset of each dimension, so that the data in the corresponding dimension can be corrected based on the offset of each dimension to obtain the adjustment result of the data in the corresponding dimension.

[0087] Furthermore, this application does not limit the implementation of step 111 above. For example, it can specifically be: finding the offset that corresponds to the state adjustment information above from a pre-built mapping relationship, and using it as the offset of the parameter corresponding to the target state. This mapping relationship is used to record the offsets corresponding to different state adjustment information, such as different adjustment intensity values ​​and / or different attitude identifiers, so that the offset corresponding to the user-specified state adjustment information can be quickly obtained subsequently using this mapping relationship.

[0088] For example, step 111 above can be specifically described as: mapping the state adjustment information above to the offset of the parameter corresponding to the target state according to the pre-set rules.

[0089] Based on the relevant content of step 111 above, in some scenarios, when the target state is body shape and the parameters corresponding to the target state include data in 10 dimensions, if the user detects the state adjustment information specified for the body shape, such as an adjustment intensity value of -0.5, then the offset of the parameters corresponding to the target state, such as the offset of each dimension, is determined according to the state adjustment information.

[0090] Step 112: Adjust the parameters corresponding to the target state according to the offset of the parameters corresponding to the target state to obtain the adjustment result corresponding to the target state.

[0091] In this application, in some scenarios, when the target state is body shape and the parameters corresponding to the target state include data in 10 dimensions, after obtaining the offset of the parameter, such as the offset corresponding to each dimension, the parameter is adjusted according to the offset to obtain the adjustment result corresponding to the target state.

[0092] Based on the relevant content of steps 111 to 112 above, in some scenarios, when the target image provided by the user is used to describe at least one candidate state of an object, if the user detects state adjustment information specified by the user for the object in the target image, and the state adjustment information is used to instruct adjustment processing for the target state among the at least one candidate state, then firstly, based on the state adjustment information, the offset of the parameter corresponding to the target state is determined so that the offset can indicate how to adjust the parameter corresponding to the target to meet the adjustment requirements described by the state adjustment information; then, based on the offset, the parameter corresponding to the target state is adjusted to obtain the adjustment result corresponding to the target state, so that the adjustment result meets the state adjustment requirements specified by the user for the target state.

[0093] Based on the relevant content of step 11 above, in some scenarios, when the target image provided by the user is used to describe the two states of an object's body shape and posture, and the state parameters of the object in the target image include parameters corresponding to the body shape and parameters corresponding to the posture, if the user-specified state adjustment information for the object is detected, and the state adjustment information is used to indicate the adjustment processing for the body shape, then firstly, based on the state adjustment information, such as adjusting the intensity value by -0.5, the offset of the parameters corresponding to the body shape is determined; then, based on the offset, the parameters corresponding to the body shape are adjusted to obtain the adjustment result corresponding to the body shape, so that the adjustment result can meet the body shape adjustment requirements described by the state adjustment information.

[0094] Step 12: Based on the adjustment results corresponding to the target state, determine the adjusted parameters, which include the adjustment results corresponding to the target state.

[0095] It should be noted that this application does not limit the implementation of step 12 above. For ease of understanding, the following description is provided in conjunction with some situations.

[0096] Scenario 1: In some scenarios, at least one candidate state mentioned above may only include the target state. Based on this, step 12 above can specifically be: determining the adjustment result corresponding to the target state as the adjusted parameter.

[0097] Scenario 2: In some scenarios, at least one candidate state mentioned above includes not only the target state but also other states. Based on this, when the state parameters of the object in the target image also include parameters corresponding to other states besides the target state among the at least one candidate state, step 12 above can specifically be: Based on the adjustment result corresponding to the target state and the parameters corresponding to the other states, determine the adjusted parameters so that the adjusted parameters include the adjustment result corresponding to the target state and the parameters corresponding to the other states, thereby ensuring that the adjusted parameters satisfy the following constraints: the target state described by the adjusted parameters is different from the target state described by the state parameter, but any other state among the at least one candidate state described by the adjusted parameters besides the target state is the same as the corresponding state described by the state parameter. This is beneficial for better meeting the user's state adjustment needs, such as adjusting only body shape.

[0098] Based on the relevant content of steps 11 to 12 above, in some scenarios, when the target image provided by the user is used to describe the body shape and posture of an object, and the state parameters of the object in the target image include parameters corresponding to the body shape and parameters corresponding to the posture, if the user-specified state adjustment information for the object is detected, and the state adjustment information is used to indicate adjustment processing for the body shape, then according to the state adjustment information, the parameters corresponding to the body shape in the state parameters are adjusted to obtain the adjusted parameters, so that the target state described by the adjusted parameters is different from the target state described by the state parameters, and so that any other state among the at least one candidate state described by the adjusted parameters, except for the target state, is the same as the corresponding state described by the state parameters. This is beneficial to better meet the user's state adjustment needs, such as adjusting only the body shape.

[0099] Research has revealed that in some scenarios, such as those using SMPL technology for state prediction, the different dimensions of the parameters corresponding to body shape are not decoupled. This means that adjusting a shape indicator of the body shape, such as the weight indicator, can also affect other shape indicators, such as the height indicator. Therefore, to overcome the problem caused by this effect, this application also provides a possible implementation of S2 above. In this implementation, when the state parameters above include parameters corresponding to the target state, and the target state is body shape, if the state adjustment information above is used to instruct adjustments to be made to other parts or all shape indicators (such as the weight indicator) of the body shape except for height, then S2 can specifically include steps 21-24 below.

[0100] Step 21: Based on the status adjustment information above, adjust the parameters corresponding to the target status to obtain the adjustment result corresponding to the target status.

[0101] It should be noted that for the relevant content of step 21, please refer to the relevant content of step 11 above.

[0102] Step 22: Based on the adjustment result corresponding to the target state, determine the adjusted height of the object so that the adjusted height can represent the height described by the adjustment result.

[0103] It should be noted that this application does not limit the implementation of step 22. For example, it can be implemented using any existing or future method that can determine height based on parameters (such as body shape parameters of SMPL), such as a pre-built machine learning model with height determination function.

[0104] In addition, to further improve accuracy, step 22 above can be specifically described as follows: First, based on the adjustment result corresponding to the target state, determine the coordinates of the object's head vertex and the coordinates of the object's feet; then calculate the distance between the coordinates of the object's head vertex and the coordinates of the object's feet to obtain the object's adjusted height, so that the adjusted height can represent the height of the 3D model with the adjustment result, thereby enabling the adjusted height to represent the height described by the adjustment result.

[0105] It should be noted that this application does not limit the implementation method of "determining the coordinates of the top of the object and the coordinates of the bottom of the object based on the adjustment result corresponding to the target state" in the above paragraph. For example, it can be implemented by any existing or future method that can determine the coordinates of some key points based on parameters (such as the body shape parameters of SMPL), such as a pre-built machine learning model with coordinate determination function.

[0106] Furthermore, to improve accuracy, when the aforementioned state parameters also include parameters corresponding to at least one candidate state other than the target state, the process for determining the "coordinates of the object's head vertex and the coordinates of the object's feet" can be as follows: First, based on the adjustment result corresponding to the target state and the parameters corresponding to the other states, determine the parameters to be used, so that the parameters to be used include the adjustment result corresponding to the target state and the parameters corresponding to the other states; then, use the parameters to be used to drive the 3D model, so that the state of the driven 3D model is consistent with the state described by the parameters to be used; then, perform image acquisition on the driven 3D model to obtain a 2D image corresponding to the adjustment result; finally, perform keypoint detection processing on the 2D image to obtain the coordinates of the object's head vertex and the coordinates of the object's feet.

[0107] Based on the relevant content of step 22 above, in some scenarios, if the target state is a body shape, after adjusting the parameters corresponding to the target state to obtain the adjustment result, the image of the 3D model with the adjustment result can be acquired first to obtain the 2D image corresponding to the adjustment result; then, key point detection processing is performed on the 2D image to obtain the key point detection result, so that the key point detection result includes the coordinates of the object's head vertex and the coordinates of the object's feet; then, based on the key point detection result, the adjusted height of the object is calculated so that the adjusted height can represent the height described by the adjustment result, so that the impact of this adjustment on the shape index of height can be determined based on the adjusted height.

[0108] Step 23: Based on the ratio between the adjusted height of the object and the original height of the object, update the adjustment result corresponding to the target state to obtain the updated result corresponding to the target state. The height described by the updated result is consistent with the original height, which is determined based on the parameters corresponding to the target state.

[0109] The original height of an object refers to its height before adjustments are made to the parameters corresponding to the target state, such as the height described by the parameters corresponding to the target state.

[0110] Therefore, in one possible implementation, the object's original height can be determined based on parameters corresponding to the target state, so that the original height can represent the object's height before adjustments were made to the parameters corresponding to the target state. It should be noted that the implementation of this original height determination process is similar to the implementation of the adjusted height determination process described above, and for the sake of brevity, it will not be repeated here.

[0111] Additionally, the phrase "the ratio between the adjusted height of an object and its original height" above describes the impact on the shape metric of height when parameters corresponding to the target state are adjusted, such as the height scaling ratio, so that this impact can be addressed based on this ratio later.

[0112] Furthermore, this application does not limit the implementation of step 23 above. For example, it can be implemented using any existing or future method that can adjust parameters based on the height scaling ratio.

[0113] Furthermore, to further improve accuracy, step 23 above can be specifically described as follows: After obtaining the adjustment result corresponding to the target state, firstly, image acquisition is performed on the 3D model with the adjustment result to obtain the 2D image corresponding to the adjustment result; then, based on the "ratio between the adjusted height of the object and the original height of the object" mentioned above, height adjustment processing is performed on the object in the 2D image, such as height enlargement or height reduction, to obtain the processed image, so that the height of the object in the processed image is consistent with the original height; finally, parameter prediction processing is performed on the processed image to obtain the update result corresponding to the target state, so that the update result can describe at least one candidate state of the object as a whole in the processed image, and the update result at least satisfies the following constraints: the height described by the update result is consistent with the original height, and the update result satisfies the body shape adjustment requirements described by the state adjustment information.

[0114] It should be noted that this application does not limit the implementation method of the height adjustment processing described above. For example, it can employ any existing or future method capable of scaling up or down an object in an image, such as using a machine learning model with height adjustment capabilities. Furthermore, this application does not limit the implementation method of the step "performing parameter prediction processing on the processed image to obtain the updated result corresponding to the target state" described above. For example, the implementation method of this step is similar to the method of obtaining the "state parameters of the object in the target image" described above, and will not be elaborated upon here for the sake of brevity.

[0115] Based on the relevant content of step 23 above, in some scenarios, if the target state is body shape, after adjusting the parameters corresponding to the target state to obtain the adjustment result, the adjustment result corresponding to the target state can be updated according to the ratio between the adjusted height of the object and the original height of the object, so as to obtain the updated result corresponding to the target state. This ensures that the height described by the updated result is consistent with the original height and that the updated result meets the body shape adjustment requirements described by the state adjustment information above. This can effectively overcome the impact of adjusting one indicator of body shape on another indicator, thereby improving the adjustment effect.

[0116] Step 24: Based on the update results corresponding to the target state, determine the adjusted parameters, which include the update results corresponding to the target state.

[0117] It should be noted that the implementation method of step 24 is similar to that of step 12 above, and will not be repeated here for the sake of brevity.

[0118] Based on the relevant content of steps 21 to 24 above, in some scenarios, when the target image provided by the user is used to describe the body shape and posture of an object, and the state parameters of the object in the target image include parameters corresponding to the body shape and parameters corresponding to the posture, if the user-specified state adjustment information for the object is detected, and this state adjustment information is used to instruct adjustments to other parts or all shape indicators (such as the fatness indicator) of the body shape except for height, then first, according to the state adjustment information, such as adjusting the intensity value -0.5, the parameters corresponding to the body shape are adjusted to obtain the adjustment result corresponding to the body shape, so that the adjustment result can meet the body shape adjustment requirements described by the state adjustment information; then, according to the ratio between the height described by the adjustment result and the height described by the parameters corresponding to the body shape, the adjustment result is updated to obtain the updated result corresponding to the target state, so that the height described by the updated result is consistent with the height described by the parameters corresponding to the body shape, and the updated result meets the body shape adjustment requirements described by the state adjustment information. This can effectively overcome the impact of adjusting one body shape indicator on another indicator, thereby improving the adjustment effect.

[0119] Based on the relevant content of S2 above, in some scenarios, such as image editing, after obtaining the state parameters of the object in the target image provided by the user, such as the SMPL parameter, if the state adjustment information specified by the user for the object with the state parameter is detected, the state parameter is adjusted according to the state adjustment information to obtain the adjusted parameter, so that the adjusted parameter meets the state adjustment requirements described by the state adjustment information, so that the state adjustment processing of the object in the target image can be realized based on the state parameter in the future.

[0120] S3: Based on the adjusted parameters and the target image, a generated image is obtained; the target state described by the generated image is consistent with the target state described by the adjusted parameters, and the other information described by the generated image besides the target state is consistent with the other information described by the target image besides the target state.

[0121] The generated image refers to the image generated based on the adjusted parameters and the target image, so that the target state described by the generated image, such as body shape, is consistent with the target state described by the adjusted parameters, and that other information described by the generated image other than the target state is consistent with other information described by the target image other than the target state. In this way, state adjustment processing can be performed on the target image while maintaining other information except for the target state.

[0122] It should be noted that this application does not limit the implementation of the concept of "maintaining consistency". For example, in some scenarios, the concept of "maintaining consistency" is used to mean that they are completely identical. As an example, if information 1 and information 2 maintain consistency, it can mean that information 1 and information 2 are completely identical.

[0123] For example, in some scenarios, the concept of "maintaining consistency" mentioned above is used to indicate a relatively high degree of similarity. As an example, if information 1 and information 2 maintain consistency, it can be said that the degree of similarity between information 1 and information 2 is relatively high, such as if the degree of similarity between information 1 and information 2 is higher than a preset similarity threshold.

[0124] Furthermore, this application does not limit the implementation of S3 above. For example, it can be implemented using any existing or future method that can generate new images based on state parameters (e.g., SMLP parameters) and existing images, such as a method implemented by means of a pre-built image generation model.

[0125] In addition, to further improve the effect, this application also provides a possible implementation of S3 above. In this implementation, S3 can specifically be: obtaining a generated image based on the state feature extraction result of the adjusted parameters, the semantic information extraction result of the target image, and the image feature extraction result of the target image.

[0126] Regarding the state feature extraction result of the adjusted parameters mentioned above, the state feature extraction result is obtained by performing state feature extraction processing on the adjusted parameters so that the state feature extraction result can represent the state described by the adjusted parameters, such as body shape and posture.

[0127] Furthermore, this application does not limit the implementation method of the state feature extraction process in the above paragraph. For example, it can adopt any existing or future method that can perform state feature extraction processing on state parameters (e.g., SMLP parameters), such as by using a machine learning model with state extraction function.

[0128] Furthermore, to better improve the state adjustment effect on the target object, this application also provides a method for determining the state feature extraction result of the adjusted parameters mentioned above. Specifically, this method involves first acquiring an image of the 3D model with the adjusted parameters to obtain a 2D image, such as... Figure 2 Image 2 is shown to enable the two-dimensional image to represent the state described by the adjusted parameters in an image-based manner. Then, feature extraction processing is performed on the two-dimensional image to obtain the state feature extraction result of the adjusted parameters. This state feature extraction result can represent the state described by the adjusted parameters in a feature map manner, so that the state feature extraction result can not only represent the state described by the adjusted parameters, but also some image information, such as spatial information. This makes the information dimension of the state feature extraction result more compatible with the information dimensions of other feature maps, such as the semantic information extraction result of the target image and the image feature extraction result of the target image. This can effectively avoid the defects caused by the mismatch of information dimensions of data from different sources, thereby improving the adjustment effect.

[0129] Furthermore, this application does not limit the implementation of the step "performing feature extraction processing on the two-dimensional image to obtain the state feature extraction result of the adjusted parameters" in the above paragraph. For example, it can be implemented using any existing or future method capable of performing feature extraction processing on an image.

[0130] For example, when S3 above is implemented using a target model, the state feature extraction module in the target model can be used to implement the step of "performing feature extraction processing on the two-dimensional image to obtain the state feature extraction result of the adjusted parameters".

[0131] The target model is used to generate images based on the input data of the target model; and the relevant content of the target model, such as the training process, can be found below.

[0132] In addition, the state feature extraction module is used to perform feature extraction processing on the input data of the state feature extraction module, such as... Figure 2 Feature extraction 2 is shown so that the state feature extraction module can be used to obtain the state feature extraction result of the adjusted parameters; and this application does not limit the implementation of the state feature extraction module. For example, the state feature extraction module can be implemented using any encoder capable of performing feature extraction processing on images.

[0133] For example, in some scenarios, the state feature extraction module can be implemented using Convolutional Neural Networks (CNNs). As an example, this state feature extraction module may include a 3×3 convolutional layer and three max pooling layers, enabling it to not only extract features from its input data but also perform 8x downsampling on the input data. Therefore, in one possible implementation, the state feature extraction module can be used to perform both feature extraction and 8x downsampling on its input data.

[0134] Regarding the image feature extraction result of the target image mentioned above, this image feature extraction result is obtained by performing image feature extraction processing on the target image, so that the image feature extraction result can represent the image information carried by the target image, such as the local information of each pixel and some global information of the target image. It can be seen that, in one possible implementation, the image feature extraction result is at least used to represent the local information of each pixel in the target image, so that the image feature extraction result can represent the detailed information carried by the target image as accurately as possible. Therefore, the image feature extraction result can at least be used to characterize the local information of different parts of the object in the target image, and thus can accurately represent the characteristics of each part of the object in the target image, such as a ring on a finger or a bracelet on a wrist.

[0135] Furthermore, this application does not limit the implementation method of the image feature extraction process in the above paragraph. For example, the image feature extraction process can be implemented using existing or future methods that can perform pixel-level information extraction processing on an image, such as the method implemented by an image encoder with pre-built image feature extraction function.

[0136] For example, when S3 above is implemented using a target model, the image feature extraction module within that target model can be used to obtain the image feature extraction results of the target image. This image feature extraction module is used to perform feature extraction processing on the input data, such as... Figure 2 Feature extraction 1 is shown; moreover, this application does not limit the implementation of the image feature extraction module. For example, the image feature extraction module can be implemented using any encoder capable of performing feature extraction processing on an image.

[0137] For example, in one possible implementation, the image feature extraction module described above can be used to perform feature extraction processing on the input data of the image feature extraction module, and to perform 8x downsampling processing on the input data, so that the output data of the image feature extraction module meets the preset size requirements, such as the size requirements of the input data of the first network described below.

[0138] Furthermore, in some scenarios, to improve accuracy, this application also provides a possible implementation of the target model described above. In this implementation, the structure of the state feature extraction module in the target model is the same as that of the image feature extraction module in the target model, to ensure that the output data of the two modules are of the same size. This avoids defects caused by inconsistent output data sizes between the two modules, such as poor image generation results. However, the parameters in the state feature extraction module are different from those in the image feature extraction module, so that the state feature extraction module is more suitable for performing state feature extraction processing on an image, and the image feature extraction module is more suitable for performing image feature extraction processing on an image, thereby giving the target model better performance.

[0139] Regarding the semantic information extraction result of the target image mentioned above, this semantic information extraction result is obtained by performing semantic information extraction processing on the target image, so that the semantic information extraction result can represent the semantic information of the target image. This allows the semantic information extraction result to more accurately and comprehensively represent the global information of the target image, such as information like image architecture, so that it can provide sufficient global information for the subsequent image generation process. Therefore, in one possible implementation, the semantic information extraction result can be used to represent the global information of the target image.

[0140] Furthermore, this application does not limit the implementation method of semantic information extraction and processing in the above paragraph. For example, it can be implemented using any existing or future method that can perform semantic information extraction and processing on images, such as by using the image encoder in the Contrastive Language-Image Pre-Training (CLIP) model.

[0141] Furthermore, to improve the results, this application also provides a method for obtaining the generated image described above, in which the generated image can be obtained using a first network and a second network. For ease of understanding, these two networks are described below.

[0142] Regarding the first network mentioned above, such as Figure 2 In the case of network 1 shown, the first network is used to process the image feature extraction results of the target image so that the first network can obtain as comprehensively and accurately as possible the information carried by the target image that needs to be retained during the image generation process, such as background and object ID.

[0143] Furthermore, this application does not limit the implementation method of the first network. For example, to improve performance, the first network may employ a UNet network, such as... Figure 2 The network 1 shown is implemented.

[0144] As can be seen, in one possible implementation, the first network may include M encoding modules corresponding to different sizes and M decoding modules corresponding to the same sizes, and these M encoding modules and M decoding modules are distributed in a U-shaped structure. The size of the output data of the encoding module corresponding to the m-th size is the m-th size, and the size of the input data of the decoding module corresponding to the m-th size is the m-th size, where m is a positive integer, m ≤ M, and M is a positive integer. For better understanding, an example is provided below.

[0145] As an example, when the first network includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the first size > the second size > the third size > ... > the Mth size, if the input data of the first network includes the image feature extraction results of the target image, then the first network at least satisfies the following constraints:

[0146] The input data of the encoding module corresponding to the first size includes the image feature extraction result of the target image; the input data of the encoding module corresponding to the second size includes the output data of the encoding module corresponding to the first size; the input data of the encoding module corresponding to the third size includes the output data of the encoding module corresponding to the second size; ... (and so on); the input data of the encoding module corresponding to the Mth size includes the output data of the encoding module corresponding to the (M-1)th size;

[0147] Furthermore, the input data of the decoding module corresponding to the Mth size includes the output data of the encoding module corresponding to the Mth size; the input data of the decoding module corresponding to the (M-1)th size includes the output data of the decoding module corresponding to the Mth size; the input data of the decoding module corresponding to the (M-2)th size includes the output data of the decoding module corresponding to the (M-1)th size; ... (and so on); the input data of the decoding module corresponding to the 1st size includes the output data of the decoding module corresponding to the 2nd size.

[0148] Furthermore, to further improve the performance of the first network, this application also provides a possible implementation of the first network. In this implementation, the first network at least satisfies the following constraint: semantic information extraction results of the target image are introduced into some or all modules of the first network, enabling these modules to perform corresponding processing with the assistance of the semantic information extraction results. This allows the first network to acquire information with the aid of the semantic information extraction results, thereby improving the information acquisition performance of the first network. It should be noted that this application does not limit the method of introducing the semantic information extraction results; for example, it can be introduced using a cross-attention approach.

[0149] Research has revealed that in some scenarios, the semantic information extraction result of the target image is obtained by encoding the target image, such as the encoding process in CLIP, so that the semantic information extraction result belongs to the encoded feature. Also, because the first part of the UNet network is used for encoding processing, when the semantic information extraction result is introduced into the first part of the module, the encoding processing will be repeated. This may increase the complexity of the first network and reduce the processing efficiency of the first network without improving the performance.

[0150] Based on the above research, in order to better simplify the first network and improve its efficiency without reducing performance, this application provides a possible implementation of the first network. In this implementation, the first network satisfies at least the following constraint: the semantic information extraction result of the target image is introduced into each decoding module of the first network so that the semantic information extraction result only affects the decoding process of the first network. This can effectively avoid the defects caused by repeatedly encoding the semantic information extraction result, thereby simplifying the first network and improving its processing efficiency.

[0151] As can be seen, in one possible implementation, the semantic information extraction result of the target image above can be injected into the decoding module of the first network through cross-attention, so that the semantic information extraction result can be used to assist the decoding process of the first network, thereby enabling the first network to acquire information better and faster with the assistance of the semantic information extraction result, which in turn helps to improve the information acquisition performance of the first network.

[0152] Based on the above two paragraphs, it can be seen that, in one possible implementation, when the first network includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the first size > the second size > the third size > ... > the Mth size, if the input data of the first network includes the image feature extraction results of the target image, then the first network can at least satisfy the following constraints:

[0153] The input data of the encoding module corresponding to the first size includes the image feature extraction result of the target image; the input data of the encoding module corresponding to the second size includes the output data of the encoding module corresponding to the first size; the input data of the encoding module corresponding to the third size includes the output data of the encoding module corresponding to the second size; ... (and so on); the input data of the encoding module corresponding to the Mth size includes the output data of the encoding module corresponding to the (M-1)th size;

[0154] Furthermore, the input data of the decoding module corresponding to the Mth size includes the output data of the encoding module corresponding to the Mth size and the semantic information extraction result of the target image above; the input data of the decoding module corresponding to the (M-1)th size includes the output data of the decoding module corresponding to the Mth size and the semantic information extraction result; the input data of the decoding module corresponding to the (M-2)th size includes the output data of the decoding module corresponding to the (M-1)th size and the semantic information extraction result; ... (and so on); the input data of the decoding module corresponding to the 1st size includes the output data of the decoding module corresponding to the 2nd size and the semantic information extraction result.

[0155] In addition, to further improve efficiency, this application also provides a possible implementation of the first network. In this implementation, when the above decoding module includes a first sub-module and a second sub-module, and the input data of the second sub-module includes the output data of the first sub-module, the first network satisfies at least the following constraint: the semantic information extraction result of the target image is injected into the second sub-module of the decoding module of the first network through a cross-attention method, so that the semantic information extraction result assists the processing involved in the second sub-module.

[0156] It can be seen that, in one possible implementation, the semantic information extraction results of the target image can be introduced into some networks within each module of the first network, so that the semantic information extraction results can assist some processing processes involved in each decoding process of the first network, which can be more conducive to simplifying the first network and improving the processing efficiency of the first network.

[0157] Regarding the second network mentioned above, such as Figure 2 For the network 2 shown, the second network is used to process the state feature extraction results of the parameters adjusted above, so that the second network can fuse the state described by the state feature extraction results, such as body shape, with other information in the target image other than the state, so as to achieve state adjustment processing while maintaining other information in the target image.

[0158] Furthermore, this application does not limit the implementation of the second network. For example, in some scenarios, the second network can be used for denoising. Thus, in one possible implementation, the second network is used to process the noisy image and the state feature extraction result of the adjusted parameters (or, the result obtained by fusing the noisy image and the state feature extraction result of the adjusted parameters) to achieve the denoising process. Here, the noisy image refers to the image that needs denoising processing; and this application does not limit the method of obtaining the noisy image. For example, the noisy image can be obtained by generating random noise.

[0159] It should be noted that this application does not limit the implementation method of fusion in the above paragraph. For example, if the noise image and the state feature extraction results of the adjusted parameters above are both implemented using feature maps, then the fusion may refer to the summation of two feature maps or the splicing of two feature maps. This application does not make specific limitations on this.

[0160] For example, to further improve performance, the second network can use a UNet network, such as... Figure 2 The network 2 shown is implemented.

[0161] Furthermore, to further improve performance, the network structure of the second network is the same as that of the first network described above. This ensures that when the output data of a module in the first network is introduced into the corresponding module in the second network, there is as little information loss as possible, which is beneficial to improving performance.

[0162] Based on the above two paragraphs, it can be seen that in one possible implementation, when the first network mentioned above includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, the second network also includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the second network at least satisfies the following constraints:

[0163] The structure of the encoding module corresponding to the m-th size in the second network is the same as the structure of the encoding module corresponding to the m-th size in the first network, and the structure of the decoding module corresponding to the m-th size in the second network is the same as the structure of the decoding module corresponding to the m-th size in the first network. m is a positive integer, m≤M. This ensures that information loss is minimized when introducing information between different modules under the same structure.

[0164] Furthermore, the parameters in the encoding module corresponding to the m-th size in the second network are different from those in the encoding module corresponding to the m-th size in the first network, and the parameters in the decoding module corresponding to the m-th size in the second network are also different from those in the decoding module corresponding to the m-th size in the first network, where m is a positive integer and m≤M. This makes each module in the second network more suitable for denoising and makes each module in the first network more suitable for acquiring image information such as background and object ID.

[0165] Furthermore, in order to better improve the performance of the second network, in one possible implementation, the second network described above shall at least satisfy the following constraint: the output data of the corresponding modules in the first network shall be introduced into some or all of the modules in the second network, so that the second network can perform better noise reduction with the assistance of the information obtained by the first network, thereby improving performance.

[0166] Research has shown that when the output data of the corresponding modules in the first network is introduced into the first part of the second network, it not only makes the second network more complex, but also the information acquired by the first network is introduced too early, resulting in a greater influence of the information acquired by the first network, such as the state. This leads to a poor state adjustment effect of the second network on the target image.

[0167] Based on the above research, in order to better improve the performance of the second network, the second network should at least satisfy the following constraints: introduce the output data of the corresponding modules in the first network into each decoding module in the second network, so that the information acquired by the first network only affects the decoding process of the second network. This can not only reduce the complexity of the second network, but also better reduce the influence of the information acquired by the first network, thereby improving the performance of the second network with as little increase in complexity as possible.

[0168] As can be seen, in one possible implementation, the output data of the decoding module of the first network can be injected into the decoding module of the second network through cross-attention, so that the second network can perform decoding processing with the assistance of the information obtained by the first network, which is beneficial to improving the performance of the second network.

[0169] Furthermore, to further improve the performance of the second network, this application also provides a possible implementation of the second network. In this implementation, the second network at least satisfies the following constraint: semantic information extraction results of the target image are introduced into some or all modules of the second network, enabling these modules to perform corresponding processing with the assistance of the semantic information extraction results. This allows the second network to perform denoising processing with the assistance of the semantic information extraction results, thereby improving the performance of the second network. It should be noted that this application does not limit the method of introducing the semantic information extraction results; for example, it can be introduced using a cross-attention approach.

[0170] Furthermore, in order to better simplify the second network and improve its efficiency without reducing performance, this application provides a possible implementation of the second network. In this implementation, the second network at least satisfies the following constraint: the semantic information extraction result of the target image is introduced into each decoding module of the second network so that the semantic information extraction result only affects the decoding process of the second network. This can effectively avoid the defects caused by repeatedly encoding the semantic information extraction result, thereby simplifying the second network and improving its processing efficiency.

[0171] As can be seen, in one possible implementation, the semantic information extraction result of the target image above can be injected into the decoding module of the second network through cross-attention, so that the semantic information extraction result can be used to assist the decoding process of the second network, thereby enabling the second network to perform denoising better with the assistance of the semantic information extraction result, which in turn helps to improve the performance of the second network.

[0172] Based on the above description of the second network, in one possible implementation, when the first network includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the second network includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the order is 1 size > 2 size > 3 size > ... > M size, if the input data of the second network includes a noisy image and the state feature extraction results of the adjusted parameters mentioned above (or, the result obtained by fusing the noisy image and the state feature extraction results of the adjusted parameters mentioned above), then the second network can at least satisfy the following constraints:

[0173] The input data of the encoding module corresponding to the first size in the second network includes the noisy image and the state feature extraction result of the adjusted parameters mentioned above (or, the result obtained by fusing the noisy image and the state feature extraction result of the adjusted parameters mentioned above).

[0174] The input data of the encoding module corresponding to the second size in the second network includes the output data of the encoding module corresponding to the first size in the second network;

[0175] The input data of the encoding module corresponding to the third size in the second network includes the output data of the encoding module corresponding to the second size in the second network;

[0176] ... (and so on)

[0177] The input data of the encoding module corresponding to the Mth size in the second network includes the output data of the encoding module corresponding to the (M-1)th size in the second network;

[0178] The input data of the decoding module corresponding to the Mth size in the second network includes the output data of the encoding module corresponding to the Mth size in the second network, the semantic information extraction result of the target image above, and the output data of the decoding module corresponding to the Mth size in the first network;

[0179] The input data of the decoding module corresponding to the M-1th size in the second network includes the output data of the decoding module corresponding to the M-th size in the second network, the semantic information extraction result, and the output data of the decoding module corresponding to the M-1th size in the first network;

[0180] The input data of the decoding module corresponding to the M-2th size in the second network includes the output data of the decoding module corresponding to the M-1th size in the second network, the semantic information extraction result, and the output data of the decoding module corresponding to the M-2th size in the first network;

[0181] ... (and so on)

[0182] The input data of the decoding module corresponding to the first size in the second network includes the output data of the decoding module corresponding to the second size in the second network, the semantic information extraction result, and the output data of the decoding module corresponding to the first size in the first network.

[0183] Furthermore, this application does not limit the specific method of introducing the two types of information into the second network. For example, the two types of information can be introduced by merging them into one type of information, so as to simultaneously introduce the two types of information into each decoding module of the second network.

[0184] Research has revealed that because the output data sizes of different decoding modules in the first network are different, the sizes of some decoding modules in the first network are inconsistent with the sizes of the semantic information extraction results of the target image. Therefore, when these two types of information are introduced at the same time, it is necessary to overcome the size inconsistency problem, which will lead to information loss. Therefore, in order to overcome this defect, different information can be introduced at different times.

[0185] The study also found that some global information of the target image, such as image architecture, is easily lost during the processing of the second network. Therefore, in order to better improve the image generation effect, the timing of introducing the semantic information extraction results of the target image should be delayed.

[0186] Based on the above two studies, this application also provides a possible implementation of the second network mentioned above. In this implementation, when the decoding module includes a first sub-module and a second sub-module, and the input data of the second sub-module includes the output data of the first sub-module, the second network can at least satisfy the following constraints: the output data of the first sub-module in the decoding module of the first network mentioned above is injected into the first sub-module in the decoding module of the second network through cross-attention; and the semantic information extraction result of the target image mentioned above is injected into the second sub-module in the decoding module of the second network through cross-attention.

[0187] It can be seen that, in one possible implementation, when the first network includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the second network includes encoding modules corresponding to M sizes and decoding modules corresponding to the M sizes, and the first size > the second size > the third size > ... > the Mth size, the second network can at least satisfy the following constraints: the output data of the first sub-module in the decoding module corresponding to the m-th size in the first network is injected into the first sub-module in the decoding module corresponding to the m-th size in the second network through cross-attention, and the semantic information extraction result of the target image above is injected into the second sub-module in the decoding module corresponding to the m-th size in the second network through cross-attention, where m is a positive integer and m≤M.

[0188] Furthermore, to further improve the performance of the second network, this application also provides a possible implementation of the aforementioned decoding module. In this implementation, the decoding module may include a first submodule, a second submodule, and an upsampling module. The input data of the upsampling module is determined based on the output data of the second submodule. The upsampling module performs upsampling processing on its input data to make the size of its output data larger than the size of its input data, ensuring that the size of the output data of the upsampling module is the same as the size of the output data of the decoding module, and that the size of the input data of the upsampling module is the same as the size of the input data of the decoding module. This delays the size adjustment process in the decoding module, effectively ensuring that the data sizes involved in the first submodules of different networks remain consistent, thereby effectively avoiding information loss due to size inconsistencies and improving the performance of the second network. It should be noted that this application does not limit the implementation of the upsampling module.

[0189] Based on the relevant content of the first and second networks mentioned above, in some scenarios, S3 can specifically be: using the first and second networks to process the state feature extraction results of the adjusted parameters, the semantic information extraction results of the target image, and the image feature extraction results of the target image to obtain the generated image.

[0190] In addition, to further improve the image generation effect, S3 above can specifically be: using the target model to process the adjusted parameters and the target image to obtain the generated image.

[0191] Furthermore, this application does not limit the implementation method of the target model mentioned above.

[0192] Research has shown that the diffusion model has good image generation capabilities, resulting in high consistency of images generated using this model.

[0193] Based on the above research, in one possible implementation, to better improve the image adjustment effect, the target model mentioned above can adopt a diffusion model, such as... Figure 2 The diffusion model shown is implemented.

[0194] In another possible implementation, the target model described above may include a state feature extraction module, a semantic information extraction module, an image feature extraction module, a first network, and a second network. Specifically, the state feature extraction module is used to obtain the state feature extraction results of the adjusted parameters; the semantic information extraction module is used to obtain the semantic information extraction results of the target image; the image feature extraction module is used to obtain the image feature extraction results of the target image; the input data of the first network includes the image feature extraction results and the semantic information extraction results; the input data of the second network includes the state feature extraction results, the semantic information extraction results, and the output data of some or all modules in the first network. It should be noted that the relevant content of each part of this target model is described above.

[0195] As can be seen, in one possible implementation, when the target model described above can be implemented using a diffusion model, the target model can be used to process the noisy image, the adjusted parameters, and the target image to obtain a generated image; moreover, the input data of the second network in the target model specifically includes the noisy image, the state feature extraction result of the adjusted parameters, the semantic information extraction result of the target image, and the output data of some or all modules in the first network, so that the second network has better denoising capabilities.

[0196] Based on the relevant content of S1 to S3 above, for the image generation method provided in this application embodiment, the target image and the state parameters of the object in the target image are first obtained; then, according to the state adjustment information specified for the object, such as the body being 0.5 times thinner, the state parameters are adjusted to obtain the adjusted parameters; then, according to the adjusted parameters and the target image, the generated image is obtained.

[0197] In this process, a 3D model with state parameters is used to display at least one candidate state of the object in the target image in 3D space, such as body shape and / or posture. This allows the state parameters to represent the overall state of the object in the target image to a certain extent. Consequently, the adjustment processing of the state parameters can adjust the overall state of the object. Furthermore, the adjusted parameters obtained based on the adjustment processing can represent the adjustment requirements for the overall state of the object. This allows the generated image obtained based on the adjusted parameters to meet the adjustment requirements, thus better satisfying the overall adjustment requirements of the object in an image. This enables the overall coordination and uniform deformation of various parts of the object, thereby improving the image adjustment effect.

[0198] Furthermore, since the state adjustment information is used to instruct adjustment processing for a target state, such as body shape, among at least one candidate state, so that the adjusted parameters obtained based on the state adjustment information can better meet the state adjustment requirements of the target state described by the state adjustment information, the target state described by the adjusted parameters is different from the target state described by the state parameters. Consequently, the target state described by the generated image based on the adjusted parameters is consistent with the target state described by the adjusted parameters, and other information described by the generated image besides the target state is consistent with other information described by the target image besides the target state. Thus, the generated image can better meet the overall adjustment requirements of an object in an image, such as the overall body shape adjustment requirements, thereby improving the image adjustment effect.

[0199] In addition, in order to improve the image generation effect, this application also provides a possible implementation of the target model mentioned above. In this way, when the target model is a diffusion model and the target model is used to process the noisy image, the adjusted parameters and the target image to obtain the generated image, the training process of the target model may include steps 31-33 below.

[0200] Step 31: Obtain a first image, a second image, target noise, the noise addition result of the second image, and the state parameters of the second image. The noise addition result is obtained by adding noise to the second image using the target noise. The state parameters of the second image are used to describe at least one candidate state of the object as a whole in the second image.

[0201] The first image refers to the image used in the current round of training to provide information other than the state.

[0202] The second image refers to the image used to provide the state during the current round of training.

[0203] In addition, to improve the training effect, the first image and the second image must at least satisfy the following constraint: the object in the first image and the object in the second image are the same object.

[0204] In addition, to better improve the state learning effect, the first image and the second image must at least satisfy the following constraint: the object in the first image and the object in the second image are the same object, but the state described by the first image is different from the state described by the second image.

[0205] Furthermore, this application does not limit the process of acquiring the first image and the second image. For example, it can be implemented using any existing or future method that can acquire two images of the same object but different states.

[0206] Furthermore, in order to better reduce the difficulty of data acquisition, this application also provides a method for acquiring the first image and the second image. In this method, when a video is used to describe the state change of an object, the acquisition process of the first image and the second image is specifically as follows: randomly extract an image from the video as the first image, and randomly extract an image from the video as the second image, so as to ensure that the object in the first image and the object in the second image are the same object.

[0207] It can be seen that, in one possible implementation, the first image and the second image at least satisfy the following constraint: the first image and the second image are extracted from the same video, so that the object in the first image and the object in the second image are the same object.

[0208] It should be noted that this application does not limit the relationship between the first image and the second image in the above two paragraphs. For example, the first image and the second image can be the same frame image in the video, or they can be different frame images.

[0209] In addition, to better improve the state learning effect, when a video is used to describe the state change of an object, the acquisition process of the first image and the second image is as follows: two images are randomly selected from the video, one of the selected images is used as the first image, and the other of the selected images is used as the second image, so as to ensure that the object in the first image and the object in the second image are the same object. However, the state described by the first image is different from the state described by the second image.

[0210] It can be seen that, in one possible implementation, the first image and the second image at least satisfy the following constraints: the first image and the second image are different images extracted from the same video, such that the object in the first image and the object in the second image are the same object, but the state described by the first image is different from the state described by the second image.

[0211] The target noise refers to the noise actually used when adding noise to the second image; and this application does not limit the method of obtaining the target noise, for example, it can be specified by the user in advance or generated randomly.

[0212] The noise-adding result of the second image is obtained by adding noise to the second image using the target noise, and this application does not limit the representation of the noise-adding result. For example, it can be represented by using the noise image method.

[0213] The state parameters of the second image are used to describe at least one candidate state of the object as a whole in the second image, such as body shape and / or posture; and the relevant content of the "state parameters of the second image" is similar to the relevant content of the "state parameters of the object in the target image" above, so for the sake of brevity, it will not be repeated here.

[0214] Based on the relevant content of step 31 above, for the current round of training, two frames are randomly extracted from a video and used as the first image and the second image, respectively; then, target noise is added to the second image to obtain the noise-added result of the second image, and SMPL parameter prediction processing is performed on the second image to obtain the state parameters of the second image, so that a new image can be generated based on the first image, the state parameters and the noise-added result.

[0215] Step 32: Use the target model to process the state parameters of the noise-added result, the first image, and the second image to obtain the predicted noise.

[0216] Among them, prediction noise refers to the noise added during prediction, such as the prediction noise output by network 2.

[0217] Furthermore, this application does not limit the implementation of step 32 above. For example, when the target model includes a state feature extraction module, a semantic information extraction module, an image feature extraction module, a first network, and a second network, step 32 can specifically be: first, using the state feature extraction module to obtain the state feature extraction result of the state parameters of the second image, using the semantic information extraction module to obtain the semantic information extraction result of the first image, and using the image feature extraction module to obtain the image feature extraction result of the first image; then, using the first network and the second network, processing the noise addition result, the state feature extraction result, the semantic information extraction result, and the image feature extraction result to obtain and output prediction noise, so that a new image can be generated by removing the prediction noise in the future.

[0218] Step 33: Based on the difference between the predicted noise and the target noise, update the target model, and return to continue executing Step 31 and subsequent steps above until the preset stopping condition is met.

[0219] The difference between the predicted noise and the target noise is used to describe the performance of the target model; and this application does not limit the calculation method of the "difference between the predicted noise and the target noise". For example, it can be implemented by any method that can measure the difference between two noises, such as L2 loss.

[0220] A preset stopping condition refers to a pre-defined condition that must be met before iterative training of the target model can be stopped. This application does not limit the preset stopping condition; for example, it may include: the target model's loss falling below a preset loss threshold. Alternatively, it may include: the rate of change of the target model's loss falling below a preset rate of change threshold. Or, it may include: the number of updates to the target model reaching a preset number threshold.

[0221] The model loss of the target model is used to characterize the performance of the target model, and the model loss is determined based on the difference between the predicted noise and the target noise. It should be noted that this application does not limit the calculation method of the model loss.

[0222] Furthermore, this application does not limit the implementation of "updating the target model" in step 33 above. For example, it can specifically be: updating all parameters within the target model.

[0223] Furthermore, in some scenarios, since some modules in the target model, such as the semantic information extraction module, already have relatively good performance, this application also provides a possible implementation of step 33 above in order to further improve efficiency. In this way, when the target model includes a state feature extraction module, a semantic information extraction module, an image feature extraction module, a first network, a second network, and a decoder, step 33 can specifically be: based on the difference between the predicted noise and the target noise, update the state feature extraction module, the image feature extraction module, the first network, and the second network in the target model, and return to continue executing step 31 above and its subsequent steps until the preset stopping condition is reached.

[0224] Here, the decoder refers to a module within the target model used for decoding data based on the output data of the second network in the target model, such as... Figure 2 The decoder shown. It is evident that the decoder satisfies at least the following constraints: the input data of the decoder includes the output data of the second network, and the decoder is used to decode the output data of the second network to obtain a new image, such as... Figure 2 Image 3 is shown.

[0225] Based on the relevant content of steps 31 to 33 above, it can be seen that in some scenarios, some or all modules in the target model can be trained through multiple rounds of iteration to make the trained target model have better performance. This results in a better state adjustment effect when the target model is used to perform the user-triggered state adjustment task, such as the task described in S1-S3 above, which is beneficial to improving the image adjustment effect.

[0226] Based on the image generation method provided in the embodiments of this application, the embodiments of this application also provide an image generation apparatus, which will be discussed below. Figure 3 Explanation and clarification will be provided. Among them, Figure 3 This is a schematic diagram of an image generation apparatus provided in an embodiment of this application. It should be noted that for technical details of the image generation apparatus provided in this embodiment, please refer to the relevant content of the image generation method above.

[0227] like Figure 3 As shown, the image generation apparatus 300 provided in this application embodiment includes:

[0228] The first acquisition unit 301 is used to acquire a target image and state parameters of objects in the target image. A three-dimensional model with the state parameters is used to display at least one candidate state of the object as a whole in the target image in three-dimensional space. The at least one candidate state includes body shape and / or posture.

[0229] The parameter adjustment unit 302 is used to adjust the state parameters according to the state adjustment information specified for the object to obtain the adjusted parameters. The state adjustment information is used to indicate the adjustment process for the target state among the at least one candidate state. The target state described by the adjusted parameters is different from the target state described by the state parameters.

[0230] The image generation unit 303 is used to generate an image based on the adjusted parameters and the target image; the target state described by the generated image is consistent with the target state described by the adjusted parameters, and the other information described by the generated image other than the target state is consistent with the other information described by the target image other than the target state.

[0231] In one possible implementation, the state parameters include parameters corresponding to the target state;

[0232] The parameter adjustment unit 302 is specifically used for: adjusting the parameters corresponding to the target state according to the state adjustment information to obtain the adjustment result corresponding to the target state; and determining the adjusted parameters according to the adjustment result corresponding to the target state, wherein the adjusted parameters include the adjustment result corresponding to the target state.

[0233] In one possible implementation, the state parameters further include parameters corresponding to other states among the at least one candidate state besides the target state; the adjusted parameters further include parameters corresponding to the other states.

[0234] In one possible implementation, the target state is body shape; the state adjustment information is used to instruct adjustments to be made to all or part of the shape parameters of the body shape, except for height.

[0235] The parameter adjustment unit 302 is further configured to: after obtaining the adjustment result corresponding to the target state, determine the adjusted height of the object based on the adjustment result corresponding to the target state; update the adjustment result corresponding to the target state based on the ratio between the adjusted height of the object and the original height of the object, and obtain the updated result corresponding to the target state, wherein the height described by the updated result is consistent with the original height, and the original height is determined based on the parameters corresponding to the target state;

[0236] The parameter adjustment unit 302 is specifically used to: determine the adjusted parameters based on the update result corresponding to the target state, wherein the adjusted parameters include the update result corresponding to the target state.

[0237] In one possible implementation, the image generation unit 303 is further configured to: acquire the state feature extraction result of the adjusted parameters, the semantic information extraction result of the target image, and the image feature extraction result of the target image;

[0238] The image generation unit 303 is specifically used to: obtain the generated image based on the state feature extraction result, the semantic information extraction result, and the image feature extraction result.

[0239] In one possible implementation, the generated image is obtained using a first network and a second network; the first network is used to process the image feature extraction result, and the semantic information extraction result is injected into the decoding module of the first network through cross-attention; the second network is used to process the state feature extraction result, and the semantic information extraction result is injected into the decoding module of the second network through cross-attention, and the output data of the decoding module of the first network is injected into the decoding module of the second network through cross-attention.

[0240] In one possible implementation, the decoding module includes a first submodule and a second submodule, the input data of the second submodule includes the output data of the first submodule; the semantic information extraction result is injected into the second submodule of the decoding module of the first network through a cross-attention method; the output data of the first submodule in the decoding module of the first network is injected into the first submodule of the decoding module of the second network through a cross-attention method; the semantic information extraction result is injected into the second submodule of the decoding module of the second network through a cross-attention method.

[0241] In one possible implementation, the second network is used to process the noisy image and the state feature extraction results.

[0242] In one possible implementation, the network structure of the first network is the same as that of the second network.

[0243] In one possible implementation, the process of determining the state feature extraction result of the adjusted parameters includes: acquiring an image of a three-dimensional model with the adjusted parameters to obtain a two-dimensional image; and performing feature extraction processing on the two-dimensional image to obtain the state feature extraction result of the adjusted parameters.

[0244] In one possible implementation, the image generation unit 303 is specifically used to: process the adjusted parameters and the target image using the target model to obtain the generated image.

[0245] In one possible implementation, the target model is used to process the noisy image, the adjusted parameters, and the target image to obtain the generated image;

[0246] The training process of the target model includes: acquiring a first image, a second image, target noise, a noise-added result of the second image, and state parameters of the second image, wherein the noise-added result is obtained by adding noise to the second image using the target noise, and the state parameters of the second image are used to describe at least one candidate state of the object as a whole in the second image; processing the noise-added result, the first image, and the state parameters of the second image using the target model to obtain predicted noise; updating the target model based on the difference between the predicted noise and the target noise, and continuing to execute the steps of acquiring the first image, the second image, the target noise, the noise-added result of the second image, and the state parameters of the second image until a preset stopping condition is reached.

[0247] In one possible implementation, the first image and the second image are extracted from the same video.

[0248] In one possible implementation, the target model includes a state feature extraction module, a semantic information extraction module, an image feature extraction module, a first network, and a second network; the state feature extraction module is used to obtain the state feature extraction result of the adjusted parameters; the semantic information extraction module is used to obtain the semantic information extraction result of the target image; the image feature extraction module is used to obtain the image feature extraction result of the target image; the input data of the first network includes the image feature extraction result and the semantic information extraction result; the input data of the second network includes the state feature extraction result, the semantic information extraction result, and the output data of some or all modules in the first network.

[0249] Based on the aforementioned content of the image generation device 300, the working principle of the image generation device 300 provided in this application is as follows: First, a target image and the state parameters of the object in the target image are acquired; then, based on the state adjustment information specified for the object, such as a 0.5-fold reduction in body size, the state parameters are adjusted to obtain the adjusted parameters; then, based on the adjusted parameters and the target image, a generated image is obtained. The three-dimensional model with state parameters is used to display at least one candidate state of the object in the target image in three-dimensional space, such as body shape and / or posture, so that the state parameters can, to a certain extent, represent the overall state of the object in the target image. This allows the adjustment processing of the state parameters to adjust the overall state of the object, and the adjusted parameters obtained from the adjustment processing can represent the adjustment requirements for the overall state of the object. Thus, the generated image obtained based on the adjusted parameters can meet these adjustment requirements, thereby better satisfying the overall adjustment requirements of an object in an image, and achieving uniform deformation of all parts of the object, thus improving the image adjustment effect. Furthermore, since the state adjustment information is used to instruct adjustment processing for a target state, such as body shape, among at least one candidate state, so that the adjusted parameters obtained based on the state adjustment information can better meet the state adjustment requirements of the target state described by the state adjustment information, the target state described by the adjusted parameters is different from the target state described by the state parameters. Consequently, the target state described by the generated image based on the adjusted parameters is consistent with the target state described by the adjusted parameters, and other information described by the generated image besides the target state is consistent with other information described by the target image besides the target state. Thus, the generated image can better meet the overall adjustment requirements of an object in an image, such as the overall body shape adjustment requirements, thereby improving the image adjustment effect.

[0250] In addition, this application embodiment also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device performs any implementation of the image generation method provided in this application embodiment.

[0251] See Figure 4 This diagram illustrates a structural schematic of an electronic device 400 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0252] like Figure 4 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. The processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0253] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0254] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by the processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0255] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0256] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the image generation method provided in this application.

[0257] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0258] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0259] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0260] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0261] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0262] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0263] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0264] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0265] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0266] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0267] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0268] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0269] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0270] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that, The method includes: A target image and state parameters of objects in the target image are obtained. A three-dimensional model with the state parameters is used to display at least one candidate state of the object as a whole in the target image in three-dimensional space. The at least one candidate state includes body shape and / or posture. Based on the state adjustment information specified for the object, the state parameters are adjusted to obtain the adjusted parameters. The state adjustment information is used to indicate the adjustment process for the target state among the at least one candidate state. The target state described by the adjusted parameters is different from the target state described by the state parameters. Based on the adjusted parameters and the target image, a generated image is obtained; the target state described by the generated image is consistent with the target state described by the adjusted parameters, and other information described by the generated image other than the target state is consistent with other information described by the target image other than the target state.

2. The method according to claim 1, characterized in that, The state parameters include parameters corresponding to the target state; The state parameters are adjusted based on the state adjustment information specified for the object to obtain the adjusted parameters, including: Based on the state adjustment information, the parameters corresponding to the target state are adjusted to obtain the adjustment result corresponding to the target state; Based on the adjustment result corresponding to the target state, the adjusted parameters are determined, and the adjusted parameters include the adjustment result corresponding to the target state.

3. The method according to claim 2, characterized in that, The state parameters also include parameters corresponding to other states among the at least one candidate state besides the target state; The adjusted parameters also include parameters corresponding to the other states.

4. The method according to claim 2, characterized in that, The target state is body shape; The status adjustment information is used to indicate adjustments to the shape indicators of the body shape, excluding height, or all of them. After obtaining the adjustment result corresponding to the target state, the method further includes: Based on the adjustment results corresponding to the target state, determine the adjusted height of the object; Based on the ratio between the adjusted height of the object and the original height of the object, the adjustment result corresponding to the target state is updated to obtain the updated result corresponding to the target state. The height described by the updated result is consistent with the original height, which is determined based on the parameters corresponding to the target state. The step of determining the adjusted parameters based on the adjustment result corresponding to the target state includes: Based on the update result corresponding to the target state, the adjusted parameters are determined, and the adjusted parameters include the update result corresponding to the target state.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the state feature extraction results of the adjusted parameters, the semantic information extraction results of the target image, and the image feature extraction results of the target image; The step of generating an image based on the adjusted parameters and the target image includes: The generated image is obtained based on the state feature extraction results, the semantic information extraction results, and the image feature extraction results.

6. The method according to claim 5, characterized in that, The generated image is obtained using a first network and a second network; The first network is used to process the image feature extraction results, and the semantic information extraction results are injected into the decoding module of the first network through cross-attention. The second network is used to process the state feature extraction result. The semantic information extraction result is injected into the decoding module of the second network through cross-attention. The output data of the decoding module of the first network is injected into the decoding module of the second network through cross-attention.

7. The method according to claim 6, characterized in that, The decoding module includes a first submodule and a second submodule, wherein the input data of the second submodule includes the output data of the first submodule; The semantic information extraction result is injected into the second sub-module of the decoding module of the first network through a cross-attention method; The output data of the first submodule in the decoding module of the first network is injected into the first submodule in the decoding module of the second network through cross-attention. The semantic information extraction result is injected into the second sub-module of the decoding module of the second network through cross-attention.

8. The method according to claim 6, characterized in that, The second network is used to process the noisy image and the state feature extraction results.

9. The method according to claim 6, characterized in that, The network structure of the first network is the same as that of the second network.

10. The method according to claim 5, characterized in that, The process of determining the state feature extraction result of the adjusted parameters includes: Image acquisition is performed on the three-dimensional model with the adjusted parameters to obtain a two-dimensional image; The two-dimensional image is subjected to feature extraction processing to obtain the state feature extraction result of the adjusted parameters.

11. The method according to claim 1, characterized in that, The step of generating an image based on the adjusted parameters and the target image includes: The adjusted parameters and the target image are processed using the target model to obtain the generated image.

12. The method according to claim 11, characterized in that, The target model is used to process the noisy image, the adjusted parameters, and the target image to obtain the generated image; The training process of the target model includes: A first image, a second image, target noise, a noise-adding result of the second image, and state parameters of the second image are obtained. The noise-adding result is obtained by adding noise to the second image using the target noise. The state parameters of the second image are used to describe at least one candidate state of the object as a whole in the second image. The target model is used to process the state parameters of the noise addition result, the first image, and the second image to obtain the predicted noise; Based on the difference between the predicted noise and the target noise, the target model is updated, and the steps of obtaining the first image, the second image, the target noise, the noise addition result of the second image, and the state parameters of the second image are continued until a preset stopping condition is reached.

13. The method according to claim 12, characterized in that, The first image and the second image were extracted from the same video.

14. The method according to claim 11, characterized in that, The target model includes a state feature extraction module, a semantic information extraction module, an image feature extraction module, a first network, and a second network; The state feature extraction module is used to obtain the state feature extraction results of the adjusted parameters; The semantic information extraction module is used to obtain the semantic information extraction results of the target image; The image feature extraction module is used to obtain the image feature extraction results of the target image; The input data of the first network includes the image feature extraction results and the semantic information extraction results; The input data of the second network includes the state feature extraction results, the semantic information extraction results, and the output data of some or all modules in the first network.

15. An image generation apparatus, characterized in that, include: The first acquisition unit is used to acquire a target image and state parameters of objects in the target image. A three-dimensional model with the state parameters is used to display at least one candidate state of the object as a whole in the target image in three-dimensional space. The at least one candidate state includes body shape and / or posture. A parameter adjustment unit is used to adjust the state parameters according to the state adjustment information specified for the object to obtain the adjusted parameters. The state adjustment information is used to indicate the adjustment process for the target state among the at least one candidate state. The target state described by the adjusted parameters is different from the target state described by the state parameters. An image generation unit is configured to generate an image based on the adjusted parameters and the target image; the target state described by the generated image is consistent with the target state described by the adjusted parameters, and other information described by the generated image other than the target state is consistent with other information described by the target image other than the target state.

16. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-14.

17. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-14.

18. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method of any one of claims 1-14.

Images