Image generation method and device, electronic equipment and computer readable medium

Abstract

The invention discloses an image generation method and device, electronic equipment and a computer readable medium. The method comprises the steps of firstly obtaining a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image and key point representation data of a to-be-processed image; carrying out information fusion processing on the data to obtain image generation condition representation data; then, performing image generation processing according to the image generation condition representation data to obtain a target image corresponding to the to-be-processed image, so that the target image and the target object description image are both used for describing the same object, and the target image and the to-be-processed image are kept consistent under at least one piece of object state representation information; therefore, the target image can represent the object replacement result for the to-be-processed image, and the purpose of carrying out object replacement processing on an existing image or an existing video can be achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, electronic device, and computer-readable medium. Background Art

[0002] In some application scenarios, it may be necessary to replace an object appearing in a video data or an image data with an object described by the object description image based on the object description image provided by the user. For ease of understanding, the following is explained with examples.

[0003] As an example, in a face-changing scenario, a face appearing in a segment of video data can be replaced with a face described by the face description image based on the face description image provided by the user, thereby achieving a face-changing effect. Summary of the invention

[0004] The present application provides an image generation method, device, electronic device, and computer-readable medium, which can effectively improve the object replacement effect.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides an image generation method, the method comprising:

[0007] Acquire a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of the to-be-processed image; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object;

[0008] Performing information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data;

[0009] Image generation processing is performed according to the image generation condition representation data to obtain a target image corresponding to the image to be processed; the target image and the target object description image are both used to describe the same object, and the target image and the image to be processed are consistent in at least one object state representation information.

[0010] In a possible implementation manner, the process of determining the image generation condition characterization data includes:

[0011] Performing feature extraction processing on the target object description image to obtain image representation features;

[0012] Performing feature extraction processing on the to-be-replaced object description image and key point representation data of the to-be-replaced object description image to obtain multi-information representation features;

[0013] Performing feature extraction processing on the key point representation data of the image to be processed to obtain key point representation features;

[0014] The image representation features, the multi-information representation features and the key point representation features are subjected to cross-attention processing to obtain the image generation condition representation data; the key point representation features are used as query features in the cross-attention processing; the multi-information representation features are used as key features in the cross-attention processing; and the image representation features are used as value features in the cross-attention processing.

[0015] In a possible implementation, the cross attention processing includes:

[0016] Performing a dot product process on the key point representation feature and the multi-information representation feature to obtain a first dot product result;

[0017] Performing a dot product process on the first dot product result and the image representation feature to obtain a second dot product result;

[0018] The image generation condition representation data is determined based on the second dot product result and the information supplement feature; the information supplement feature is determined based on the target object description image, the multi-information representation feature, the key point representation feature, and the first dot product result.

[0019] In a possible implementation manner, the process of determining the information supplementary feature includes:

[0020] Splicing the target object description image with the multi-information representation feature to obtain a first splicing feature;

[0021] Performing a dot product process on the first concatenated feature and the first dot product result to obtain a third dot product result;

[0022] Concatenate the third dot product result with the key point representation feature to obtain a second concatenated feature;

[0023] The information supplement feature is determined according to the second splicing feature.

[0024] In a possible implementation manner, the step of splicing the target object description image with the multi-information representation feature to obtain a first splicing feature includes:

[0025] The target object description image, the key point representation data of the target object description image and the multi-information representation feature are spliced ​​to obtain a first splicing feature.

[0026] In a possible implementation manner, the target image corresponding to the image to be processed is determined based on the image generation condition characterization data and the noise data corresponding to the image to be processed;

[0027] If the image generation method is used to train some or all modules in an image generation model, the noise data is obtained by performing noise processing on the image to be processed;

[0028] or,

[0029] If the image generation method is used to perform object replacement processing on one or more images to be processed according to the target object description image, the noise data is randomly generated noise data.

[0030] In a possible implementation manner, the image generation method is used to perform object replacement processing on a to-be-processed video according to the target object description image;

[0031] The image to be processed is used to represent any video frame in the video to be processed;

[0032] Each video frame in the video to be processed corresponds to the same noise data.

[0033] In a possible implementation manner, the image generation method is used to perform object replacement processing on a video to be processed according to the target object description image; the video to be processed is determined according to a video specified operation;

[0034] The target object description image is determined according to the image specifying operation;

[0035] The object description image to be replaced refers to a video frame existing in the video to be processed and achieving the object representation performance target;

[0036] The image to be processed is used to represent any video frame in the video to be processed.

[0037] In a possible implementation manner, the method further includes:

[0038] After obtaining the target images corresponding to at least two video frames in the video to be processed, the target images corresponding to the at least two video frames are arranged and processed according to the arrangement sequence numbers corresponding to the at least two video frames in the video to be processed to obtain the target video corresponding to the video to be processed.

[0039] In a possible implementation manner, the image generation method is used to train part or all of the modules in the image generation model based on at least one image sample group;

[0040] For any image sample group, each image in the image sample group is used to describe the same object;

[0041] The target object description image, the to-be-replaced object description image, and the to-be-processed image are determined from the at least one image sample group.

[0042] In a possible implementation manner, the target object description image, the to-be-replaced object description image, and the to-be-processed image are all determined from the same image sample group;

[0043] or,

[0044] The target object description image is determined from a first image sample group, the to-be-replaced object description image and the to-be-processed image are both determined from a second image sample group, the object described by the first image sample group is different from the object described by the second image sample group, and the at least one image sample group includes the first image sample group and the second image sample group.

[0045] In a possible implementation manner, if the target object description image, the to-be-replaced object description image, and the to-be-processed image are all determined from the same image sample group, the to-be-replaced object description image is determined based on the target object description image.

[0046] In a possible implementation manner, the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image are subjected to information fusion processing to obtain image generation condition representation data, including:

[0047] Using a conditional representation module in an image generation model, the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image are processed to obtain the image generation conditional representation data;

[0048] The performing image generation processing according to the image generation condition representation data to obtain a target image corresponding to the image to be processed includes:

[0049] The image generation module in the image generation model is used to process the image generation condition representation data to obtain a target image corresponding to the image to be processed.

[0050] In a possible implementation manner, after the image generation condition representation data is processed by the image generation module in the image generation model to obtain a target image corresponding to the image to be processed, the method further includes:

[0051] Based on the difference representation data between the target image and the label image corresponding to the image to be processed, the image generation model is updated, and the step of using the conditional representation module in the image generation model to process the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed is returned to continue executing until a preset stop condition is reached.

[0052] In a possible implementation manner, updating the image generation model according to the difference representation data between the target image and the label image corresponding to the image to be processed includes:

[0053] Based on the difference representation data between the target image and the label image corresponding to the image to be processed, other modules in the image generation model except the pre-trained module are updated; the pre-trained module is used to represent the pre-trained modules existing in the image generation model; the pre-trained module is the conditional representation module or the image generation module.

[0054] In a possible implementation manner, the target object description image, the to-be-replaced object description image, and the to-be-processed image are all used to describe the same object;

[0055] The updating of the image generation model according to the difference representation data between the target image and the label image corresponding to the image to be processed includes:

[0056] The image generation model is updated according to the difference representation data between the target image and the image to be processed.

[0057] The present application provides an image generation device, comprising:

[0058] an acquisition unit, configured to acquire a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of the to-be-processed image; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object;

[0059] a fusion unit, configured to perform information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data;

[0060] A generation unit is used to perform image generation processing based on the image generation condition representation data to obtain a target image corresponding to the image to be processed; the target image and the target object description image are both used to describe the same object, and the target image and the image to be processed are consistent in at least one object state representation information.

[0061] The present application provides an electronic device, the device comprising: a processor and a memory;

[0062] The memory is used to store instructions or computer programs;

[0063] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the image generation method provided in the present application.

[0064] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the image generation method provided by the present application.

[0065] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the image generation method provided by the present application.

[0066] Compared with the related art, this application has at least the following advantages:

[0067] In the technical solution provided by the present application, a target object description image (for example, image data input by a user for describing the target object), a to-be-replaced object description image (for example, a video frame in video data that needs to be processed for object replacement and that can better represent the characteristics of the to-be-replaced object), key point representation data of the to-be-replaced object description image, and key point representation data of the to-be-processed image (for example, any one video frame in the video data) are first obtained; then, information fusion processing is performed on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data, so that the image generation condition representation data is The condition characterization data can characterize the information required for reference when performing image generation processing (for example, the characteristics of the target object itself, the state of the object to be replaced in the image to be processed, etc.); then, image generation processing is performed according to the image generation condition characterization data to obtain a target image corresponding to the image to be processed, so that the target image and the target object description image are used to describe the same object, and the target image and the image to be processed are consistent in at least one object state characterization information (for example, object posture information, etc.), so that the target image can represent the object replacement result for the image to be processed, thereby achieving the purpose of object replacement processing for an existing image or an existing video. Among them, because the image generation condition representation data is determined based on multiple information (for example, the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed), the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing image generation processing, thereby effectively avoiding defects caused by insufficient or inaccurate reference information when performing image generation processing, and thus helping to improve the object replacement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 A flowchart of an image generation method provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of a process for determining a conditional characterization feature provided in an embodiment of the present application;

[0071] Figure 3A schematic diagram of a cross-attention processing provided in an embodiment of the present application;

[0072] Figure 4 A schematic diagram of another process for determining conditional characterization features provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of another cross-attention processing provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0075] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] It is found that in some application scenarios, the object replacement process can be realized with the help of the diffusion model, and the implementation process can be specifically as follows: if you want to perform object replacement processing on a certain data (for example, a certain image data or a certain video data, etc.), after obtaining the object description image provided by the user, the corresponding module in the diffusion model can first extract the conditional features from the object description image, so that the conditional features can represent the object information described by the object description image; and then other modules in the diffusion model perform image generation processing based on the conditional features to obtain the object replacement result corresponding to the data. Among them, because the conditional features are determined based on the single information of the object description image, the information that can be represented by the conditional features may have problems such as less information and / or certain information deviations, so that the information that can be referred to when performing image generation processing based on the conditional features may be scarce or inaccurate, which leads to certain defects in this implementation scheme (for example, the overall image quality of the object replacement result is relatively poor, and the boundary part of the replaced object is prone to blurring and artifacts, etc.), thus affecting the object replacement effect.

[0077] Based on the above research, in order to better improve the object replacement effect, the present application provides an image generation method, which includes: first obtaining a target object description image (for example, image data input by a user to describe the target object), an object description image to be replaced (for example, a video frame in the video data that needs to be processed for object replacement that can better represent the characteristics of the object to be replaced), key point representation data of the image to be replaced, and key point representation data of the image to be processed (for example, any video frame in the video data); then performing information fusion processing on the target object description image, the object description image to be replaced, the key point representation data of the image to be replaced, and the key point representation data of the image to be processed to obtain an image generation method. Conditional characterization data is generated so that the image generation conditional characterization data can characterize the information required for reference when performing image generation processing (for example, the characteristics of the target object itself, the state of the object to be replaced in the image to be processed, etc.); then, image generation processing is performed according to the image generation conditional characterization data to obtain a target image corresponding to the image to be processed, so that the target image and the target object description image are used to describe the same object, and the target image and the image to be processed are consistent in at least one object state characterization information (for example, object posture information, etc.), so that the target image can represent the object replacement result for the image to be processed, thereby achieving the purpose of object replacement processing for an existing image or an existing video. Among them, because the image generation condition representation data is determined based on multiple information (for example, the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed), the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing image generation processing, thereby effectively avoiding defects caused by insufficient or inaccurate reference information when performing image generation processing, and thus helping to improve the object replacement effect.

[0078] In addition, the present application does not limit the execution subject of the image generation method provided in the embodiment of the present application. For example, the image generation method provided in the embodiment of the present application can be applied to a terminal device or a server. For another example, the image generation method provided in the embodiment of the present application can also be implemented with the help of the data interaction process between the terminal device and the server. Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a stand-alone server, a cluster server, or a cloud server.

[0079] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] In order to better understand the technical solution provided by the present application, the image generation method provided by the present application is described below with reference to some drawings. Figure 1 As shown, the image generation method provided in the embodiment of the present application includes the following S1-S3. Figure 1 A flowchart of an image generation method provided in an embodiment of the present application.

[0081] S1: Obtain a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of an image to be processed; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object.

[0082] Among them, the target object describes the image (for example, Figure 2 The image 1) shown is used to describe a target object, which refers to an object that needs to be referred to when performing object replacement on an image data or a video data.

[0083] It should be noted that the present application does not limit the implementation methods of the object. For example, in some application scenarios, the object may refer to any object that can change state (for example, a person, a robot, a virtual person, an animal, an object, etc.). In another example, in some application scenarios, the object may refer to a certain part (for example, a face, a leg, one or more components of an object, etc.).

[0084] In addition, the present application does not limit the implementation method of the target object description image mentioned above. For ease of understanding, two situations are combined for explanation below.

[0085] Case 1: When the image generation method provided in the present application is used to implement object replacement processing (for example, face replacement processing, etc.) for an existing image or an existing video, that is, when the image generation method is used to complete an object replacement task, the target object description image refers to image data provided by the user in a certain manner.

[0086] It can be seen that in a possible implementation mode, when the image generation method provided by the present application is used to perform object replacement processing on the video to be processed according to the target object description image, the above target object description image can be determined according to the image designation operation. Among them, the video to be processed refers to the video data that needs to be processed for object replacement processing; and the present application does not limit the video to be processed. For example, in some application scenarios, the video to be processed can be determined according to the video designation operation; the video designation operation refers to the operation required to be triggered when the video data is designated; and the present application does not limit the video designation operation. For example, the video designation operation can refer to the operation of selecting a certain video data from some video data. For another example, the video designation operation can also refer to the operation of inputting video data by the user with the help of a certain input device. The image designation operation refers to the operation required to be triggered when the image data is designated, and the present application does not limit the image designation operation. For example, the image designation operation can refer to the operation of selecting a certain image from some images. For another example, the image designation operation can also refer to the operation of inputting image data by the user with the help of a certain input device.

[0087] Case 2: When the image generation method provided by the present application is used to train an image generation model that can realize object replacement processing, that is, when the image generation method is used to complete the model training task, if the image generation model is trained using at least one image sample group, the target object description image mentioned above may refer to an image randomly selected from a certain image sample group. Wherein, for any image sample group, each image in the image sample group is used to describe the same object; and the present application does not limit the implementation method of the image sample group. For example, in some application scenarios, all images in the image sample group are extracted from the same sample video.

[0088] It should be noted that the present application does not limit the acquisition process of at least one image sample group in the above paragraph. For example, in some application scenarios, for the i-th image sample group, the acquisition process of the i-th image sample group can be specifically as follows: after acquiring the i-th sample video, some video frames can be extracted from the i-th sample video, and these extracted video frames are taken as a group to obtain the i-th image sample group, so that the i-th image sample group can include some or all of the video frames in the i-th sample video. Among them, because all video frames in the i-th sample video are used to describe the i-th sample object, all images in the i-th image sample group constructed based on the i-th sample video are also used to describe the i-th sample object, so that the i-th image sample group can describe the state change process of the i-th sample object. i is a positive integer, i≤I, I is a positive integer, and I represents the number of sample videos.

[0089] It should also be noted that, in the above paragraph, sample video refers to the video data required for reference in the training process of the image generation model; sample object refers to the object described by a sample video; image sample group refers to a set of images constructed based on the sample video, so that the objects described by images in the same image sample group are the same, and the images described by images in different image sample groups may be the same or different. This application does not make any specific limitations on this.

[0090] The image describing the object to be replaced (e.g. Figure 2 The image 2) shown is used to describe the object to be replaced, which refers to the object that needs to be replaced when performing object replacement on an image data or a video data, that is, the object described by the image data or the video data itself.

[0091] In addition, the present application does not limit the implementation method of the above-mentioned image describing the object to be replaced. For ease of understanding, two situations are described below.

[0092] Case 1, when the image generation method provided by the present application is used to perform object replacement processing on the video to be processed based on the target object description image, the above-mentioned description image of the object to be replaced may refer to a video frame existing in the video to be processed that reaches the object representation performance target, so that the description image of the object to be replaced can describe the object to be replaced as accurately as possible. Among them, the object representation performance target refers to a pre-set constraint required when screening video frames that can better describe the object to be replaced (for example, the face is relatively complete, the face angle is relatively positive, etc.); and the present application does not limit the implementation method of the object representation performance target. For example, the object representation performance target can be implemented using pre-set screening rules. For example, in some application scenarios, after calculating the object representation performance data of each video frame in the video to be processed under the object to be replaced, the object representation performance target can be: the object representation performance data is not lower than a preset threshold. The object representation performance data of the jth video frame is used to represent the state of the jth video frame in terms of the object representation of the object to be replaced, so that the object representation performance data of the jth video frame can indicate whether the jth video frame can represent the object to be replaced as accurately and comprehensively as possible. j is a positive integer, j≤J, J is a positive integer, and J represents the number of video frames in the video to be processed.

[0093] Case 2: When the image generation method provided by the present application is used to train an image generation model capable of implementing object replacement processing, if the image generation model is trained using at least one image sample group, the at least one image sample group includes a first image sample group and a second image sample group, the object described by the first image sample group is different from the object described by the second image sample group, and the above target object description image refers to an image randomly selected from the first image sample group, then the above to-be-replaced object description image may refer to an image randomly selected from the second image sample group (especially, an image with better object representation performance). Wherein, because the object described by the first image sample group is different from the object described by the second image sample group, the object described by the target object description image is different from the object described by the to-be-replaced object description image. The first image sample group refers to an image sample group used to provide the target object description image, and the first image sample group may refer to any one of the at least one image sample group. The second image sample group refers to an image sample group used to provide an image describing the object to be replaced; and the present application does not limit the implementation method of the second image sample group. For example, in some application scenarios (for example, scenarios where no image sample group is used to describe the same state change process of different objects), the second image sample group may refer to any other image sample group in the at least one image sample group except the first image sample group. For another example, in some application scenarios, the second image sample group may refer to an image sample group that exists in the at least one image sample group and has an associated relationship with the first image sample group. For two image sample groups that have an associated relationship in the at least one image sample group, the two image sample groups are used to describe different objects respectively, but the object state changes described by the two image sample groups remain consistent (for example, the object states described by the images at the same position in the two image sample groups are the same, etc.).

[0094] It can be seen that in some application scenarios, if at least one image sample group includes a first image sample group and a second image sample group with an associated relationship, an image can be extracted from the first image sample group as the target object description image, and an image can be extracted from the second image sample group as the object description image to be replaced. Wherein, for the first image sample group and the second image sample group with an associated relationship, the image at the kth arrangement position in the first image sample group and the image at the kth arrangement position in the second image sample group are consistent in at least one object state representation information, so that the image at the kth arrangement position in the first image sample group and the image at the kth arrangement position in the second image sample group can represent different objects in the same state (for example, expression state, body posture, etc.), so that the only difference between the two images is that the objects are different, k is a positive integer, k≤K, K is a positive integer, and K represents the number of images in the first image sample group. Among them, the object state representation information is used to represent the state of the object; and the present application does not limit the object state representation information. For example, if the object is a face, the object state representation information may include expression state, facial orientation, facial background, etc.

[0095] In fact, in some application scenarios, in order to overcome the defect of small amount of training data due to the high difficulty in obtaining model training data, the present application also provides a possible implementation of the above-mentioned image describing the object to be replaced. Under this implementation, when the image generation method provided by the present application is used to train an image generation model that can realize object replacement processing, if the image generation model is trained using at least one image sample group, then the image describing the object to be replaced and the above-mentioned target object description image can be extracted from the same image sample group, so that in the same round of training, the image describing the object to be replaced and the target object description image are used to represent the same object, thereby making it unnecessary to obtain image sample pairs with associated relationships for the image generation model, thereby effectively overcoming the defect of small amount of training data due to the high difficulty in obtaining image sample pairs with associated relationships. In this way, the defect of poor model training effect caused by small amount of training data can be effectively overcome, which is conducive to improving the model training effect.

[0096] It should be noted that the present application does not limit the acquisition process of the to-be-replaced object description image described in the previous paragraph. For example, in some application scenarios, for a certain round of training process, when the above target object description image is extracted from the i-th image sample group, the to-be-replaced object description image may refer to an image existing in the i-th image sample group that achieves the object representation performance target. For another example, in some application scenarios, in order to better improve efficiency, the to-be-replaced object description image may be determined based on the target object description image so that the to-be-replaced object description image and the target object description image belong to the same image. i is a positive integer, i≤I, and I is a positive integer.

[0097] It can be seen that in one possible implementation, when the image generation method provided in the present application is used to train an image generation model that can realize object replacement processing, if the image generation model is trained using at least one image sample group, then the above-mentioned image describing the object to be replaced and the above-mentioned image describing the target object can be the same image extracted from the same image sample group (for example, an image that achieves the object representation performance target, etc.), which is conducive to improving efficiency.

[0098] The key point representation data of the image describing the object to be replaced is used to represent the pixel distribution state in the image describing the object to be replaced, so that the key point representation data can represent the state of the object to be replaced in the image describing the object to be replaced, such as the state of the internal area of ​​the object to be replaced, the state of the peripheral area of ​​the object to be replaced, etc.

[0099] It should be noted that the present application does not limit the implementation method of the key points. For example, when the object is a face, the key points may refer to some points that are evenly distributed in the inner area and the outer area of ​​the face.

[0100] In addition, the present application does not limit the method for obtaining the above-mentioned key point representation data. For example, it can be implemented by any existing or future method that can perform key point extraction processing on an image data (for example, with the help of a pre-built machine learning model that can perform key point extraction processing on image data, etc.).

[0101] It can be seen that, under a possible implementation mode, the method of obtaining the key point representation data of the above-mentioned image describing the object to be replaced can be specifically: inputting the image describing the object to be replaced into a pre-constructed key point extraction model, so that the key point extraction model can perform key point extraction processing on the image describing the object to be replaced, and obtain the key point representation data of the image describing the object to be replaced, so that the key point representation data can represent some key points in the image describing the object to be replaced, thereby enabling the key point representation data to represent the pixel distribution state in the image describing the object to be replaced.

[0102] In addition, the present application does not limit the representation method of the key point representation data above, for example, it can be represented by digital means. For another example, it can be represented by image means.

[0103] The image to be processed refers to an image that needs to be processed by object replacement. For example, the image to be processed may refer to Figure 2 Image 3 shown.

[0104] In addition, for the above image to be processed, the image to be processed and the above image describing the object to be replaced are both used to describe the same object. It can be seen that the image to be processed is also used to describe the object to be replaced.

[0105] In addition, the present application does not limit the implementation method of the above-mentioned image to be processed. For ease of understanding, the following description is combined with three cases.

[0106] Case 1: When the image generation method provided in the present application is used to perform object replacement processing on a video to be processed based on a target object description image, the image to be processed can be used to represent any video frame in the video to be processed, so that the object replacement processing process for the image to be processed can be used to implement the object replacement processing process for each video frame in the video to be processed, thereby completing the object replacement processing for the video to be processed.

[0107] Case 2: When the image generation method provided in the present application is used to train an image generation model capable of realizing object replacement processing, if the image generation model is trained using at least one image sample group, the target object description image mentioned above refers to an image randomly selected from the first image sample group, and the image to be replaced mentioned above refers to an image randomly selected from the second image sample group (especially, an image with better object representation performance), then the image to be processed mentioned above may refer to an image randomly selected from the second image sample group; and the present application does not limit the association relationship between the image to be processed and the image to be replaced, for example, the two may be the same image or different images.

[0108] Based on the above content, it can be known that in some application scenarios, when the image generation method provided by the present application is used to train an image generation model that can realize object replacement processing, if the image generation model is trained using at least one image sample group, the above target object description image, the above object description image to be replaced, and the above image to be processed are determined from two image sample groups, and the two image sample groups are used to describe the same state changes of different objects. For example, in some application scenarios, the target object description image can be determined from the first image sample group, the object description image to be replaced and the image to be processed are both determined from the second image sample group, and the object described by the first image sample group is different from the object described by the second image sample group.

[0109] Case 3: When the image generation method provided in the present application is used to train an image generation model that can realize object replacement processing, if the image generation model is trained using at least one image sample group, and the target object description image and the to-be-replaced object description image are extracted from the same image sample group, then the above-mentioned image to be processed may refer to an image randomly selected from the image sample group, so that the target object description image, the to-be-replaced object description image, and the to-be-processed image are all used to describe the same object. This is helpful in overcoming the defect of poor model training effect caused by a small amount of training data, thereby helping to improve the model training effect.

[0110] It should be noted that the present application does not limit the implementation method of the image to be processed described in the previous paragraph. For example, in some application scenarios (for example, scenarios where the model training data is sufficient), when the target object description image above and the object description image to be replaced above are the same image extracted from the same image sample group, the image to be processed may refer to any other image in the image sample group except the image, which is conducive to improving the training effect.

[0111] Based on the above two paragraphs, it can be known that in some application scenarios, when the image generation method provided in the present application is used to train an image generation model that can realize object replacement processing, if the image generation model is trained using at least one image sample group, then the above target object description image, the above object description image to be replaced, and the above image to be processed are all determined from the same image sample group, so that the target object description image, the object description image to be replaced, and the image to be processed are all used to describe the same object, which is conducive to overcoming the defect of poor model training effect caused by the small amount of training data, thereby helping to improve the model training effect.

[0112] The key point representation data of the image to be processed is used to represent the pixel distribution state in the image to be processed, such as the state of the internal area of ​​the object to be processed, the state of the peripheral area of ​​the object to be processed, etc., so that the key point representation data can represent the state of the object to be replaced in the image to be processed; and the method of obtaining the key point representation data of the image to be processed is similar to the method of obtaining the "key point representation data of the image describing the object to be replaced" mentioned above. For the sake of brevity, it will not be repeated here.

[0113] Based on the relevant content of S1 above, it can be known that in some application scenarios, if you want to perform object replacement processing on the image to be processed, you can obtain the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed, so that you can subsequently complete the object replacement processing for the image to be processed based on these data, which is conducive to improving the object replacement effect.

[0114] S2: performing information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data.

[0115] The image generation condition representation data is used to represent the information required for reference when performing object replacement processing on the image to be processed, that is, the information required for reference when performing image generation processing on the target image corresponding to the image to be processed below.

[0116] In addition, the present application does not limit the implementation method of S2 above. For example, it can be implemented by any existing or future method that can fuse multiple data.

[0117] For another example, in some application scenarios, when the key point representation data above is represented by image data, S2 above can specifically be: performing image fusion processing on the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed to obtain image generation condition representation data. It should be noted that the present application does not limit the implementation method of the image fusion processing. For example, it can be implemented by any existing or future method that can fuse multiple image data.

[0118] In fact, in order to better improve the information fusion effect, the present application also provides a possible implementation of the above S2. Under this implementation, the S2 may specifically include the following steps 11 to 14.

[0119] Step 11: Perform feature extraction processing on the target object description image to obtain image representation features.

[0120] The image representation feature is used to represent the object information carried by the target object description image above, so that the image representation feature can represent the relevant information of the target object described by the target object description image. For example, when the target object description image is Figure 2 When the image 1 is shown, the image characterization feature may refer to Figure 2 The image encoding feature F3 is shown.

[0121] In addition, the present application does not limit the implementation method of the above step 11. For example, the step 11 can be implemented using any existing or future method that can perform feature extraction and processing on an image data (for example, a machine learning model or module with image encoding function, etc.).

[0122] It can be seen that in a possible implementation mode, the above step 11 may specifically be: encoding the target object description image (for example, Figure 2 The encoding process 3) shown is used to obtain image representation features, so that the image representation features can represent the encoding results of the target object description image, thereby enabling the image representation features to represent the object information carried by the target object description image.

[0123] Step 12: Perform feature extraction processing on the description image of the object to be replaced and the key point representation data of the description image of the object to be replaced to obtain multi-information representation features.

[0124] The multi-information characterization feature is used to represent the information carried by the above-mentioned description image of the object to be replaced and the key point characterization data of the description image of the object to be replaced, so that the multi-information characterization feature can not only represent the object information carried by the description image of the object to be replaced (for example, information related to the object to be replaced), but also represent the pixel distribution state in the description image of the object to be replaced. For example, when the description image of the object to be replaced is Figure 2 In the case of the image 2 shown in FIG. 2 , the multi-information characterization feature may refer to Figure 2 The multi-information encoding feature F1 is shown.

[0125] In addition, the present application does not limit the implementation method of the above step 12. For example, the step 12 can be implemented using any existing or future method that can perform feature extraction and processing on an image data and its key point representation data (for example, a machine learning model or module with multi-information encoding function, etc.).

[0126] It can be seen that in a possible implementation manner, the above step 12 can be specifically as follows: after obtaining the above description image of the object to be replaced and the key point representation data of the description image of the object to be replaced, first combine the description image of the object to be replaced and the key point representation data of the description image of the object to be replaced to obtain combined data; then encode the combined data (for example, Figure 2 The encoding process 1) shown in the figure obtains a multi-information characterization feature so that the multi-information characterization feature represents the encoding result for the combined data, thereby enabling the multi-information characterization feature to represent the information carried by the image describing the object to be replaced and its key point characterization data.

[0127] Step 13: Perform feature extraction on the key point representation data of the image to be processed to obtain key point representation features.

[0128] The key point representation feature is used to represent the information carried by the key point representation data of the above image to be processed, so that the key point representation feature can represent the pixel distribution state in the image to be processed. For example, when the image to be processed is Figure 2 In the case of the image 3 shown, the image characterization feature may refer to Figure 2 The keypoints shown encode feature F2.

[0129] In addition, the present application does not limit the implementation method of the above step 13. For example, the step 13 can be implemented using any existing or future method that can perform feature extraction and processing on key point representation data of an image data (for example, a machine learning model or module with key point encoding function, etc.).

[0130] It can be seen that in a possible implementation manner, the above step 13 can specifically be: encoding the key point representation data of the image to be processed (for example, Figure 2 The encoding process 2) shown is used to obtain key point characterization features, so that the key point characterization features represent the encoding results for the key point characterization data, thereby enabling the key point characterization features to represent the information carried by the key point characterization data.

[0131] Step 14: Perform cross-attention processing on the above image representation features, the above multi-information representation features and the above key point representation features to obtain image generation condition representation data; the key point representation features are used as query features in the cross-attention processing; the multi-information representation features are used as key features in the cross-attention processing; the image representation features are used as value features in the cross-attention processing.

[0132] In the present application, after obtaining the above image representation feature, the above multi-information representation feature and the above key point representation feature, the image representation feature can be used as a value feature (for example, Figure 2-Figure 5 The Value feature shown in the figure), using the multi-information representation feature as the key feature (for example, Figure 2-Figure 5 The Key feature shown in the figure), and using the key point representation feature as the query feature (for example, Figure 2-Figure 5 The Query feature shown in the figure is used for cross-attention processing to obtain image generation condition representation data, so that the image generation condition representation data can represent the information carried by the above target object description image, the above object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the above image to be processed, so that the image generation condition representation data can represent the image information presented after the object replacement processing is performed on the image to be processed to a certain extent, and then the image generation condition representation data can represent the prototype of the object replacement processing result for the image to be processed to a certain extent. It can be seen that because the image generation condition representation data is determined based on a variety of information (for example, the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed), the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing image generation processing, so as to effectively avoid the defects caused by the lack or inaccuracy of reference information, and thus help to improve the object replacement effect.

[0133] In addition, the present application does not limit the implementation of the above cross attention processing. For example, it can be implemented by any existing or future cross attention method. For ease of understanding, a possible implementation of step 14 above is used as an example for explanation.

[0134] As an example, in a possible implementation, the above step 14 may specifically include the following steps 141 and 142.

[0135] Step 141: perform dot multiplication processing on the above key point representation feature and the above multi-information representation feature to obtain a first dot multiplication result.

[0136] The first dot product result refers to the dot product processing result between the key point representation feature and the multi-information representation feature. For example, when the key point representation feature is Figure 3 The key point encoding feature F2 shown, and the multi-information representation feature is Figure 3 When the multi-information encoding feature F1 is shown, the first point multiplication result may refer to Figure 3 The dot product attention processing result D3 is shown.

[0137] In addition, the present application does not limit the implementation method of the above step 141. For example, in some application scenarios, the step 141 may specifically be: performing a dot multiplication attention process on the above key point representation feature and the above multi-information representation feature (for example, Figure 3 The dot product attention processing shown in FIG. 1 is performed to obtain the first dot product result. It should be noted that the present application is not limited to the dot product attention processing. For example, it can be implemented by any existing or future dot product attention processing method.

[0138] For another example, in some application scenarios, in order to improve the data processing effect (eg, data processing efficiency), the above step 141 may specifically include the following steps 1411 to 1413.

[0139] Step 1411: compress and linearly process the key point representation features above to obtain key point processed features.

[0140] The key point processed feature refers to the compressed and linear processing result of the key point representation feature above, so that the key point processed feature can represent the information carried by the key point representation data of the above image to be processed with as little data as possible. For example, when the key point representation feature is Figure 3 When the key point encoding feature F2 is shown, the feature after processing of the key point can be Figure 3 The feature D1 shown has a size of q×d. q represents the size of the feature D1 in one dimension (for example, the height dimension), and d represents the size of the feature D1 in another dimension (for example, the width dimension).

[0141] In addition, the present application does not limit the implementation methods of compression and linear processing. For example, it may include any one or more compression processing processes and any one or more linear processing processes.

[0142] Step 1412: compress and linearly process the above multi-information representation features to obtain multi-information processed features.

[0143] The multi-information processed feature refers to the compressed and linear processing result of the multi-information representation feature above, so that the multi-information processed feature can represent the information carried by the key point representation data of the above to-be-replaced object description image and the to-be-replaced object description image with as little data as possible. For example, when the multi-information representation feature is Figure 3 When the multi-information encoding feature F1 is shown, the multi-information processed feature can be Figure 3 The feature D2 shown has a size of k×d. k represents the size of the feature D2 in one dimension, and d represents the size of the feature D2 in another dimension.

[0144] Step 1413: Perform dot multiplication on the features after the key point processing above and the features after the multi-information processing above to obtain a first dot multiplication result.

[0145] In this application, after obtaining the above key point processed features (for example, Figure 3 The feature D1 with a size of q×d as shown above and the feature after multi-information processing (for example, Figure 3 After the feature D2 having a size of k×d as shown, the feature after key point processing and the feature after multi-information processing can be processed by point multiplication (for example, Figure 3 The dot product attention process shown in the figure) is used to obtain the first dot product result (for example, Figure 3 The characteristic D3 shown has q×k).

[0146] Based on the relevant content of steps 1411 to 1413 above, it can be known that in some application scenarios, after obtaining the above key point representation features and the above multi-information representation features, some preset processing (for example, compression processing and linear processing, etc.) can be performed on these two features respectively, and then the processing results corresponding to these two features are subjected to dot product processing to obtain the first dot product result.

[0147] Based on the relevant content of step 141 above, it can be known that in some application scenarios, after obtaining the above key point representation features and the above multi-information representation features, dot product attention processing can be performed on these two features to obtain a first dot product result, so that the above image generation condition representation data can be determined based on the first dot product result.

[0148] Step 142: Perform a dot multiplication process on the first dot product result and the image representation feature to obtain image generation condition representation data.

[0149] In the present application, after obtaining the first dot product result above, the first dot product result can be directly dot-multiplied with the above image representation feature to obtain image generation condition representation data, thereby completing the cross-attention processing process between the above image representation feature, the above multi-information representation feature and the above key point representation feature.

[0150] In fact, in some application scenarios, in order to better improve the data processing effect (for example, data processing efficiency), the present application also provides a possible implementation of the above step 142. In this implementation, the step 142 can be specifically: firstly, the above image representation features (for example, Figure 3 The image coding feature F3 shown in the figure is compressed to obtain the image compression feature (for example, Figure 3The feature D4 with a size of k×d' as shown above) is used to enable the image compression feature to represent the object information carried by the target object description image above with as little data as possible; then the first point multiplication result above (for example, Figure 3 The feature D3 with q×k as shown in the figure is multiplied with the image compression feature to obtain the image generation condition representation data (for example, Figure 3 The feature D3 shown has a size of q×d'). Wherein, d' represents the size value of the feature D4 in one dimension.

[0151] Based on the relevant contents of steps 141 to 142 above, it can be known that for some application scenarios, after obtaining the above image representation features, the above multi-information representation features and the above key point representation features, the key point representation features can be used as query features and the multi-information representation features can be used as key features for dot multiplication processing; the dot multiplication processing result is then dot multiplied with the image representation feature as the value feature to obtain image generation condition representation data, so as to complete the cross-attention processing process between the above image representation features, the above multi-information representation features and the above key point representation features (for example, Figure 3 The crossed attention process shown).

[0152] In fact, in order to better improve the image generation effect, the present application also provides a possible implementation of the above step 14. Under this implementation, the step 14 may specifically include the following steps 143 to 145.

[0153] Step 143: perform dot multiplication processing on the above key point representation feature and the above multi-information representation feature to obtain a first dot multiplication result.

[0154] It should be noted that for the relevant content of step 143, please refer to the relevant content of step 141 above, and for the sake of brevity, it will not be repeated here.

[0155] Step 144: perform dot product processing on the first dot product result and the image representation feature to obtain a second dot product result.

[0156] The second dot product result refers to the dot product processing result between the first dot product result and the image representation feature.

[0157] In addition, the implementation of step 144 is similar to the implementation of step 142 above, and for ease of understanding, it is not repeated here.

[0158] Step 145: Determine the image generation condition representation data based on the second dot product result and the information supplement feature; the information supplement feature is determined based on the target object description image, the multi-information representation feature, the key point representation feature, and the first dot product result.

[0159] Among them, the information supplement feature refers to the feature required to supplement the information described by the second dot product result above; and the information supplement feature is determined based on the target object description image above, the multi-information representation feature above, the key point representation feature above, and the first dot product result above, so that the information supplement feature can represent the information represented by the target object description image, the multi-information representation feature, the key point representation feature and the first dot product result.

[0160] In addition, the present application does not limit the determination process of the above-mentioned information supplementary features. For example, it can be implemented by any existing or future method that can fuse multiple information.

[0161] In fact, in order to better improve the image generation effect, the present application also provides a possible implementation method of the process of determining the above-mentioned information supplementary features. Under this implementation method, the process of determining the information supplementary features may specifically include the following steps 1451-1454.

[0162] Step 1451: splice the above target object description image and the above multi-information representation feature to obtain a first spliced ​​feature.

[0163] Among them, the first splicing feature refers to the splicing result between the above target object description image and the above multi-information representation feature, so that the first splicing feature can represent the information carried by the target object description image and the multi-information representation feature, so that the first splicing feature can represent the information carried by the target object description image, the above to-be-replaced object description image and the key point representation data of the to-be-replaced object description image.

[0164] In addition, the present application does not limit the implementation of the above step 1451. For example, in some application scenarios, the step 1451 may specifically be: firstly describe the above target object image (for example, Figure 5 The image 1 shown in FIG. 1 is compressed to obtain a compressed image (for example, Figure 5 The data D5 having a size of k×3 as shown above is characterized by the multi-information (e.g., Figure 5 The multi-information encoding feature F1 shown in the figure is compressed and linearly processed to obtain the multi-information processed feature (for example, Figure 5The feature D2 having a size of k×d as shown in the figure); the compressed image is then spliced ​​with the feature after multi-information processing to obtain a first spliced ​​feature (for example, a spliced ​​feature having a size of k×(d+3)), so that the first spliced ​​feature can represent the information carried by the key point representation data of the target object description image, the above-mentioned image of the object to be replaced and the image of the object to be replaced.

[0165] In addition, in order to better improve the comprehensiveness of information supplementation, the present application also provides a possible implementation method of the above step 1451. Under this implementation method, the step 1451 can specifically be: splicing the above target object description image, the key point representation data of the target object description image, and the above multi-information representation feature to obtain a first splicing feature, so that the first splicing feature can represent the information carried by the target object description image, the key point representation data of the target object description image, and the multi-information representation feature, so that the first splicing feature can represent the information carried by the target object description image, the key point representation data of the target object description image, the above to-be-replaced object description image, and the key point representation data of the to-be-replaced object description image.

[0166] It can be seen that, under a possible implementation mode, the above step 1451 can be specifically as follows: first compress the above target object description image to obtain a compressed image, compress the key point representation data of the target object description image to obtain compressed data, and compress and linearly process the multi-information representation feature to obtain a multi-information processed feature; then splice the compressed image, the compressed data and the multi-information processed feature to obtain a first spliced ​​feature, so that the first spliced ​​feature can represent the information carried by the target object description image, the key point representation data of the target object description image, the above object description image to be replaced and the key point representation data of the object description image to be replaced.

[0167] Step 1452: Perform a dot product process on the first concatenated feature and the first dot product result to obtain a third dot product result.

[0168] The third dot product result refers to the dot product result between the first splicing feature and the first dot product result. For example, when the first splicing feature is a splicing feature with a size of k×(d+3), and the first dot product result is Figure 5 When the feature D3 with q×k is shown, the third dot product result can be Figure 5 Feature D6 is shown having a size of q×(d+3).

[0169] Step 1453: concatenate the third dot product result and the key point representation feature to obtain a second concatenated feature.

[0170] The second splicing feature refers to the splicing processing result between the third dot product result and the key point characterization feature, so that the second splicing feature can represent the information carried by the third dot product result and the key point characterization feature.

[0171] In addition, the present application does not limit the implementation method of the above step 1453. For example, in some application scenarios, the step 1453 may specifically be: firstly characterize the above key point features (for example, Figure 5 The key point encoding feature F2 shown in the figure is compressed and linearly processed to obtain the key point processed feature (for example, Figure 5 The feature D1 with the size of q×d as shown); then the feature after the key point processing is multiplied with the third point multiplication result above (for example, Figure 5 The feature D6) with a size of q×(d+3) as shown is spliced ​​to obtain a second spliced ​​feature (for example, a data feature with a size of q×(d+3+d)).

[0172] Step 1454: Determine the supplementary feature of the above information based on the second splicing feature of the above information.

[0173] It should be noted that the present application does not limit the implementation method of the above step 1454. For example, it can be specifically: pre-processing the above second splicing feature to obtain an information supplement feature so that the size of the information supplement feature is consistent with the size of the above second dot product result (for example, if the size of the second dot product result is q×d', the size of the information supplement feature is also q×d'). The pre-processing can be set in advance according to the actual application scenario. For example, in some application scenarios, the pre-processing can be implemented using a multilayer perceptron (MLP).

[0174] Based on the relevant contents of steps 1451 to 1454 above, it can be known that in some application scenarios, the information supplement feature can be determined based on the target object description image above, the multi-information representation feature above, the key point representation feature above, and the first dot product result above, so that the information supplement feature can represent the information represented by the target object description image, the multi-information representation feature, the key point representation feature and the first dot product result.

[0175] In addition, the present application does not limit the implementation method of the above step 145. For example, it can be specifically: adding the above second dot product result and the information supplement feature to obtain image generation condition representation data, so that the image generation condition representation data carries the information described by the second dot product result and the information supplement feature, so that the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing object replacement processing on the image to be processed, so as to achieve the purpose of using the information represented by the information supplement feature to supplement and correct the information represented by the second dot product result, thereby effectively avoiding the defects caused by incomplete or inaccurate information represented by the second dot product result, which is beneficial to improving the object replacement effect.

[0176] Based on the relevant contents of steps 143 to 145 above, it can be known that in some application scenarios, the image generation condition representation data can be determined based on the above image representation features, the above multi-information representation features, the above key point representation features and the above target object description image, so that the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing object replacement processing on the image to be processed. Among them, because the target object description image can provide some information that is lost (and / or deviated) when determining the image representation features, the image generation condition representation data determined based on these four data can more accurately and comprehensively represent the information required for reference when performing object replacement processing on the image to be processed, thereby helping to improve the image generation effect.

[0177] Based on the relevant contents of steps 11 to 14 above, it can be known that for some application scenarios, after obtaining the above target object description image, the above object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the above image to be processed, feature extraction processing can be performed on these data first; then, attention processing is performed on these extracted features to obtain image generation condition representation data, so that the image generation condition representation data can better represent the information required for reference when performing object replacement processing on the image to be processed, which is conducive to improving the object replacement effect.

[0178] Based on the relevant content of S2 above, it can be known that for some application scenarios, after obtaining the above target object description image, the above object description image to be replaced, the key point representation data of the image to be replaced and the key point representation data of the above image to be processed, these data can be fused in a certain way to obtain image generation condition representation data, so that the image generation condition representation data can better represent the information required for reference when performing object replacement processing on the image to be processed, which is conducive to improving the object replacement effect.

[0179] S3: Perform image generation processing according to the image generation condition representation data to obtain a target image corresponding to the image to be processed; the target image and the target object description image are both used to describe the same object, and the target image is consistent with the image to be processed in at least one object state representation information.

[0180] The target image refers to the result obtained by performing object replacement processing on the above image to be processed, so that the target image can represent the image information presented after the object to be replaced in the image to be processed is replaced with the target object. It can be seen that the target image and the above target object description image are used to describe the same object (that is, the target object), and the target image and the image to be processed are consistent in at least one object state representation information (such as expression state, body posture, etc.).

[0181] In addition, the present application does not limit the above process of determining the target image. For example, in a possible implementation, the process of determining the target image may include the following steps 21 and 22.

[0182] Step 21: Obtain the noise data corresponding to the above image to be processed.

[0183] The noise data corresponding to the image to be processed refers to the noise data required to be used when performing object replacement processing on the image to be processed.

[0184] In addition, the present application does not limit the implementation method of the noise data corresponding to the image to be processed above. For ease of understanding, three implementation methods are described below.

[0185] In a possible implementation, in order to improve efficiency as much as possible, when the image generation method provided by the present application is used to perform object replacement processing on one or more images to be processed according to the target object description image described above, the noise data corresponding to the image to be processed can be randomly generated noise data (for example, randomly generated pure noise). Wherein, because the image generation condition characterization data described above is determined based on the image to be processed above, so that the image generation condition characterization data can characterize the information carried by the image to be processed to a certain extent, so that the information carried by the image to be processed can no longer be considered in the process of generating noise data, so that the resource overhead caused by the noise generation processing based on the image to be processed can be effectively avoided, thereby effectively reducing the resource overhead of the object replacement process.

[0186] In addition, in some application scenarios (for example, scenarios such as object replacement processing for video data), in order to better improve efficiency, the present application also provides a possible implementation method of the above-mentioned noise data. In this implementation method, when the image generation method provided by the present application is used to perform object replacement processing on the video to be processed based on the above-mentioned target object description image, and the above-mentioned image to be processed is used to represent any video frame in the video to be processed, each video frame in the video to be processed corresponds to the same noise data, so that the noise data involved in the object replacement processing of each video frame in the video to be processed is the same noise data. This can effectively avoid the resource overhead caused by performing noise generation processing on each video frame separately, thereby helping to reduce the resource overhead of the object replacement process.

[0187] In addition, in some application scenarios, in order to maximize the object exchange effect, the present application also provides a possible implementation of the above-mentioned noise data. In this implementation, when the image generation method provided by the present application is used to perform object replacement processing on one or more images to be processed based on the above-mentioned target object description image, the noise data corresponding to the image to be processed can be obtained by adding noise to the image to be processed, so that the noise data can describe the object information carried by the image to be processed to a certain extent, so that the image generation process based on the noise data can better utilize the object information carried by the image to be processed, which is conducive to improving the object replacement effect.

[0188] In addition, in one possible implementation, when the image generation method provided in the present application is used to train an image generation model that can realize object replacement processing, the noise data corresponding to the above-mentioned image to be processed is obtained by adding noise to the image to be processed.

[0189] Step 22: Based on the above image generation condition characterization data, image generation processing is performed on the noise data corresponding to the above image to be processed to obtain a target image corresponding to the image to be processed.

[0190] It should be noted that the present application does not limit the implementation method of step 22. For example, step 22 can be implemented with the help of a module existing in the diffusion model that can perform image generation processing based on conditional characterization data and noise data.

[0191] Based on the relevant contents of steps 21 to 22 above, it can be known that in some application scenarios, after obtaining the noise data corresponding to the above image to be processed and the above image generation condition characterization data, image generation processing can be performed based on these two data to obtain the target image corresponding to the image to be processed, so that the target image can represent the object replacement processing result for the image to be processed.

[0192] Based on the relevant contents of S1 to S3 above, it can be known that for the image generation method provided in the embodiment of the present application, first obtain the target object description image (for example, image data input by the user for describing the target object), the to-be-replaced object description image (for example, a video frame in the video data that needs to be replaced and can better represent the characteristics of the to-be-replaced object), the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image (for example, any one video frame in the video data); then perform information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain the image generation conditions. Characterization data, so that the image generation condition characterization data can characterize the information required for reference when performing image generation processing (for example, the characteristics of the target object itself, the state of the object to be replaced in the image to be processed, etc.); then, image generation processing is performed according to the image generation condition characterization data to obtain a target image corresponding to the image to be processed, so that the target image and the target object description image are used to describe the same object, and the target image and the image to be processed are consistent in at least one object state characterization information (for example, object posture information, etc.), so that the target image can represent the object replacement result for the image to be processed, thereby achieving the purpose of object replacement processing for an existing image or an existing video. Among them, because the image generation condition representation data is determined based on multiple information (for example, the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed), the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing image generation processing, thereby effectively avoiding defects caused by insufficient or inaccurate reference information when performing image generation processing, and thus helping to improve the object replacement effect.

[0193] In fact, for the image generation method provided in the embodiment of the present application, the image generation method can be used not only to implement object replacement processing for an existing image, but also to implement object replacement processing for an existing video, and the object replacement processing can at least include the following step 31.

[0194] Step 31: After obtaining the target images corresponding to at least two video frames in the above video to be processed, the target images corresponding to the at least two video frames are arranged and processed according to the arrangement sequence numbers corresponding to the at least two video frames in the video to be processed to obtain the target video corresponding to the video to be processed.

[0195] Among them, the jth video frame refers to the video frame existing in the above video to be processed and in the jth arrangement position; and the target image corresponding to the jth video frame refers to the result obtained by performing object replacement processing on the jth video frame in the video to be processed; and the determination process of the target image corresponding to the jth video frame is similar to the determination process of the target image shown in S1-S3 above, and for the sake of brevity, it is not repeated here. j is a positive integer, j≤J, J is a positive integer, and J represents the number of video frames in the video to be processed.

[0196] The target video corresponding to the above-mentioned video to be processed refers to the result obtained by performing object replacement processing on the video to be processed, so that the object described by each video frame in the target video is the target object, and each video frame in the target video is consistent with the corresponding video frame in the video to be processed in at least one object state representation information.

[0197] Based on the relevant content of step 31 above, it can be known that in some application scenarios, for the above video to be processed, each video frame in the video to be processed can be used as the above image to be processed, and the object replacement processing for the video frame is completed by means of the image generation process shown in S1-S3 above, so that the purpose of object replacement processing for an existing video can be achieved. For ease of understanding, a possible implementation method of the image generation method is used as an example for explanation below.

[0198] As an example, in some application scenarios, the above image generation method may specifically include the following steps 41 to 44.

[0199] Step 41: In response to an image specifying operation, a target object description image is determined according to the image specifying operation.

[0200] It should be noted that for the relevant contents of step 41, please refer to the relevant contents of the above image designation operation.

[0201] Step 42: In response to the video designation operation, a video to be processed is determined according to the video designation operation, and a video frame that reaches the object representation performance target is selected from the video to be processed as the object description image to be replaced.

[0202] It should be noted that for the relevant content of step 42, please refer to the relevant content of the specified operation in the above video.

[0203] Step 43: For any video frame in the above video to be processed, the video frame is used as the image to be processed, and the target image corresponding to the video frame is obtained by executing the above S2-S3.

[0204] It should be noted that for the relevant content of step 43, please refer to the relevant content of S2-S3 above.

[0205] Step 44: According to the arrangement sequence numbers corresponding to the video frames in the above video to be processed, the target images corresponding to all the video frames are arranged and processed to obtain the target video corresponding to the video to be processed, so that the target video can represent the object replacement result for the video to be processed.

[0206] It should be noted that for the relevant content of step 44, please refer to the relevant content of step 31 above.

[0207] Based on the relevant contents of steps 41 to 44 above, it can be known that for some application scenarios, any implementation of the image generation method provided in the present application can be used to implement object replacement processing for an image data or a video data. Among them, because the image generation method has good image generation performance, the object replacement processing result determined by the image generation method for an image data or a video data has high quality, which is conducive to improving the object replacement effect.

[0208] In fact, in some application scenarios, in order to better improve the object replacement effect, some or all steps in the above image generation method can be implemented with the help of a machine learning model. For ease of understanding, the following is an example.

[0209] As an example, the above image generation method may include the following steps 51 to 53.

[0210] Step 51: Obtain a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of an image to be processed; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object.

[0211] It should be noted that the relevant contents of step 51 can be found in the relevant contents of S1 above, and for the sake of brevity, they will not be repeated here.

[0212] Step 52: Use the conditional representation module in the image generation model to process the above target object description image, the above to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the above to-be-processed image to obtain image generation conditional representation data.

[0213] The image generation model is used to perform image generation processing on the input data of the image generation model; and the present application does not limit the implementation method of the image generation model. For example, it can be implemented using a diffusion model.

[0214] In addition, the present application does not limit the construction method of the above image generation model. For example, it can be constructed according to the actual application scenario. For example, in some application scenarios, in order to reduce the difficulty of model construction, the image generation model can be constructed by replacing the text conditional generation module in the text image generation model based on the diffusion model with the following conditional representation module, so that the constructed image generation model includes the conditional representation module and the following image generation module, so that the constructed image generation model can be used to perform object replacement processing tasks.

[0215] The conditional characterization module is used to perform conditional characterization data extraction processing on the input data of the conditional characterization module; and the present application does not limit the working principle of the conditional characterization module. For example, the working principle of the conditional characterization module may include Figure 2 and Figure 3 As another example, the working principle of the condition characterization module may include: Figure 4 and Figure 5 The data processing process shown.

[0216] Step 53: Use the image generation module in the above image generation model to process the above image generation condition representation data to obtain the target image corresponding to the above image to be processed.

[0217] Among them, the image generation module is used to perform image generation processing on the input data of the image generation module; and the present application does not limit the implementation method of the image generation module. For example, it can be implemented using any existing or future module that can perform image generation processing based on conditional features and noise data (for example, the structure other than the text conditional generation module in the text image generation model based on the diffusion model).

[0218] Based on the relevant contents of steps 51 to 53 above, it can be known that in some application scenarios, the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image can be first obtained; these data are then input into the above image generation model, so that the conditional representation module in the image generation model determines the image generation conditional representation data based on these data, and the image generation module in the image generation model processes the corresponding noise data based on the image generation conditional representation data to obtain the target image corresponding to the to-be-processed image, so that the object replacement processing can be realized with the help of the image generation model.

[0219] In fact, in some application scenarios, the present application also provides some possible implementation methods of the training process of the above image generation model. For ease of understanding, they are explained below with examples.

[0220] As an example, the training process of the above image generation model may include the following steps 61 to 64.

[0221] Step 61: Obtain a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of an image to be processed; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object.

[0222] It should be noted that the relevant contents of step 61 can be found in the relevant contents of S1 above, and for the sake of brevity, they will not be repeated here.

[0223] Step 62: Use the conditional representation module in the image generation model to process the above target object description image, the above to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the above to-be-processed image to obtain image generation conditional representation data.

[0224] It should be noted that for the relevant content of step 62, please refer to the relevant content of step 52 above.

[0225] Step 63: Use the image generation module in the above image generation model to process the above image generation condition representation data to obtain the target image corresponding to the above image to be processed.

[0226] It should be noted that for the relevant content of step 63, please refer to the relevant content of step 53 above.

[0227] Step 64: Update the image generation model based on the difference representation data between the target image and the label image corresponding to the image to be processed, and return to continue executing step 62 and subsequent steps until a preset stop condition is reached.

[0228] The label image corresponding to the image to be processed is used to represent the theoretical value of the object replacement processing result for the image to be processed, so that the label image can be used as guiding information to influence the model updating process.

[0229] In addition, the present application does not limit the implementation method of the label image corresponding to the above-mentioned image to be processed. For ease of understanding, some situations are described below.

[0230] Case 1, when the above image generation model is trained using at least one image sample group, the at least one image sample group includes a first image sample group and a second image sample group with an associated relationship, the object described by the first image sample group is different from the object described by the second image sample group, and each image in the first image sample group is consistent with the corresponding image in the second image sample group in at least one object state representation information, for any round of training process, if the target object description image involved in this round of training process is an image randomly selected from the first image sample group, and the object description image to be replaced and the image to be processed involved in this round of training process are both images selected from the second image sample group, then the label image corresponding to the image to be processed involved in this round of training process may refer to an image existing in the first image sample group and corresponding to the position of the image to be processed in the second image sample group. For ease of understanding, the following is an explanation with examples.

[0231] As an example, when the image at the kth arrangement position in the first image sample group is consistent with the image at the kth arrangement position in the second image sample group in at least one object state representation information, k is a positive integer, k≤K, K is a positive integer, and K represents the number of images in the first image sample group, if the target object description image above is the image at the yth arrangement position in the first image sample group, and the image to be processed above refers to the image at the xth arrangement position in the second image sample group, then the label image corresponding to the image to be processed may refer to the image at the xth arrangement position in the first image sample group. Wherein y is a positive integer and x is a positive integer.

[0232] Case 2: If the above image generation model is trained using at least one image sample group, and the above target object description image, the above object description image to be replaced, and the above image to be processed are all determined from the same image sample group, then the label image corresponding to the image to be processed can be the image to be processed itself.

[0233] Based on the content of the above paragraph, it can be known that in one possible implementation, when the above target object description image, the above object description image to be replaced and the above image to be processed are all used to describe the same object, the above step 64 can specifically be: based on the difference representation data between the above target image and the above image to be processed, update the image generation model, and return to continue executing the above step 62 and its subsequent steps until the preset stop condition is reached.

[0234] The above difference characterization data is used to characterize the difference between two images; and the present application does not limit the implementation method of the difference characterization data. For example, it can be implemented using any existing or future method that can measure the difference between two images.

[0235] The preset stop condition refers to a pre-set condition that needs to be met when the training process for the image generation model ends; and the present application does not limit the preset stop condition. For example, the preset stop condition may include: the model loss of the image generation model is lower than a first threshold. For another example, the preset stop condition may include: the rate of change of the model loss of the image generation model is lower than a second threshold. For another example, the preset stop condition may include: the number of updates of the image generation model reaches a third threshold. Among them, the model loss of the image generation model is used to represent the image generation performance of the image generation model; and the present application does not limit the determination process of the model loss. For example, it can be implemented using any existing or future method that can calculate the loss based on the above difference characterization data.

[0236] In fact, in some application scenarios, for the above image generation model, since some modules in the image generation model have completed pre-training processing so that the module has better data processing performance, in order to improve training efficiency, the module may not be updated during the update process of the image generation model.

[0237] Based on the above content, in order to better improve the training efficiency, the present application also provides a possible implementation of the above step 64. Under this implementation, the step 64 can be specifically as follows: based on the difference representation data between the above target image and the label image corresponding to the above to-be-processed image, update the other modules in the image generation model except the pre-training module. The pre-training module is used to represent the modules that exist in the image generation model and have been pre-trained; and the present application does not limit the implementation of the pre-training module, for example, the pre-training module is a conditional representation module or an image generation module.

[0238] It can be seen that in one possible implementation, when the conditional representation module in the above image generation model has completed the pre-training processing in advance, the above step 64 can be specifically: based on the difference representation data between the above target image and the label image corresponding to the above image to be processed, update the other modules in the image generation model except the conditional representation module.

[0239] In another possible implementation, when the image generation module in the above image generation model has completed the pre-training process in advance, the above step 64 can specifically be: based on the difference representation data between the above target image and the label image corresponding to the above image to be processed, update the other modules in the image generation model except the image generation module.

[0240] Based on the relevant contents of steps 61 to 64 above, it can be known that in some application scenarios, for an image generation model including a conditional representation module and an image generation module, the image generation model can be trained using at least one image sample group, so that in a round of training for the image generation model, the target object description image, the object description image to be replaced and the image to be processed required for the current round can be first selected from these image sample groups; then, the image generation model determines the target image corresponding to the image to be processed based on the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced and the key point representation data of the image to be processed; then, based on the difference representation data between the target image and the label image corresponding to the image to be processed, some or all of the modules in the image generation model are updated, so that the next round of training can be carried out based on the updated image generation model until the preset stop condition is reached.

[0241] In addition, after completing the training process for the above image generation model, the trained image generation model has better image generation performance (for example, object replacement function), so the trained image generation model can be used to perform object replacement tasks (for example, the object replacement tasks shown in steps 41-44 above, etc.) subsequently.

[0242] Based on the image generation method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation device. Figure 6 Explain and illustrate. Figure 6 This is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. It should be noted that for the technical details of the image generating device provided in an embodiment of the present application, please refer to the relevant contents of the image generating method above.

[0243] like Figure 6 As shown, the image generating device 600 provided in the embodiment of the present application includes:

[0244] An acquisition unit 601 is used to acquire a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of an image to be processed; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object;

[0245] A fusion unit 602 is used to perform information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data;

[0246] A generation unit 603 is used to perform image generation processing based on the image generation condition representation data to obtain a target image corresponding to the image to be processed; the target image and the target object description image are both used to describe the same object, and the target image and the image to be processed are consistent in at least one object state representation information.

[0247] In a possible implementation manner, the fusion unit 602 includes:

[0248] A first extraction subunit is used to perform feature extraction processing on the target object description image to obtain image representation features;

[0249] A second extraction subunit is used to perform feature extraction processing on the description image of the object to be replaced and the key point representation data of the description image of the object to be replaced to obtain a multi-information representation feature;

[0250] A third extraction subunit is used to perform feature extraction processing on the key point representation data of the image to be processed to obtain key point representation features;

[0251] A cross-attention subunit is used to perform cross-attention processing on the image representation features, the multi-information representation features and the key point representation features to obtain the image generation condition representation data; the key point representation features are used as query features in the cross-attention processing; the multi-information representation features are used as key features in the cross-attention processing; and the image representation features are used as value features in the cross-attention processing.

[0252] In a possible implementation manner, the cross attention subunit includes:

[0253] A first processing subunit, configured to perform a dot product process on the key point representation feature and the multi-information representation feature to obtain a first dot product result;

[0254] A second processing subunit is used to perform a dot product process on the first dot product result and the image representation feature to obtain a second dot product result;

[0255] The first determination subunit is used to determine the image generation condition representation data based on the second dot product result and the information supplement feature; the information supplement feature is determined based on the target object description image, the multi-information representation feature, the key point representation feature, and the first dot product result.

[0256] In a possible implementation manner, the cross attention subunit further includes:

[0257] A third processing subunit is used to perform splicing processing on the target object description image and the multi-information representation feature to obtain a first splicing feature;

[0258] a fourth processing subunit, configured to perform a dot product process on the first concatenation feature and the first dot product result to obtain a third dot product result;

[0259] A fifth processing subunit, configured to perform a splicing process on the third dot product result and the key point representation feature to obtain a second splicing feature;

[0260] The second determining subunit is used to determine the information supplement feature according to the second splicing feature.

[0261] In a possible implementation manner, the third processing subunit is specifically used to: perform splicing processing on the target object description image, the key point representation data of the target object description image, and the multi-information representation feature to obtain a first splicing feature.

[0262] In a possible implementation manner, the target image corresponding to the image to be processed is determined based on the image generation condition characterization data and the noise data corresponding to the image to be processed;

[0263] If the image generation device 600 is used to train some or all modules in the image generation model, the noise data is obtained by performing noise processing on the image to be processed;

[0264] or,

[0265] If the image generating device 600 is used to perform object replacement processing on one or more images to be processed according to the target object description image, the noise data is randomly generated noise data.

[0266] In a possible implementation manner, the image generating device 600 is used to perform object replacement processing on the video to be processed according to the target object description image;

[0267] The image to be processed is used to represent any video frame in the video to be processed;

[0268] Each video frame in the video to be processed corresponds to the same noise data.

[0269] In a possible implementation manner, the image generating device 600 is used to perform object replacement processing on a video to be processed according to the target object description image; the video to be processed is determined according to a video designated operation;

[0270] The target object description image is determined according to the image specifying operation;

[0271] The object description image to be replaced refers to a video frame existing in the video to be processed and achieving the object representation performance target;

[0272] The image to be processed is used to represent any video frame in the video to be processed.

[0273] In a possible implementation manner, the image generating device 600 further includes:

[0274] An arrangement unit is used to arrange the target images corresponding to at least two video frames in the video to be processed according to the arrangement numbers corresponding to the at least two video frames in the video to be processed after obtaining the target images corresponding to the at least two video frames in the video to be processed, so as to obtain the target video corresponding to the video to be processed.

[0275] In a possible implementation manner, the image generation device 600 is used to train part or all of the modules in the image generation model according to at least one image sample group;

[0276] For any image sample group, each image in the image sample group is used to describe the same object;

[0277] The target object description image, the to-be-replaced object description image, and the to-be-processed image are determined from the at least one image sample group.

[0278] In a possible implementation manner, the target object description image, the to-be-replaced object description image, and the to-be-processed image are all determined from the same image sample group;

[0279] or,

[0280] The target object description image is determined from a first image sample group, the to-be-replaced object description image and the to-be-processed image are both determined from a second image sample group, the object described by the first image sample group is different from the object described by the second image sample group, and the at least one image sample group includes the first image sample group and the second image sample group.

[0281] In a possible implementation manner, if the target object description image, the to-be-replaced object description image, and the to-be-processed image are all determined from the same image sample group, the to-be-replaced object description image is determined based on the target object description image.

[0282] In a possible implementation manner, the fusion unit 602 is specifically used to: use the conditional representation module in the image generation model to process the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain the image generation condition representation data;

[0283] The generating unit 603 is specifically used to: process the image generation condition representation data using the image generation module in the image generation model to obtain a target image corresponding to the image to be processed.

[0284] In a possible implementation manner, the image generating device 600 further includes:

[0285] An updating unit is used to update the image generation model according to the difference representation data between the target image and the label image corresponding to the image to be processed, and return to the fusion unit 602 to continue to execute the step of using the conditional representation module in the image generation model to process the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed until a preset stop condition is reached.

[0286] In one possible implementation, the updating unit is specifically used to: update other modules in the image generation model except the pre-trained module based on the difference representation data between the target image and the label image corresponding to the image to be processed; the pre-trained module is used to represent the pre-trained modules existing in the image generation model; the pre-trained module is the conditional representation module or the image generation module.

[0287] In a possible implementation manner, the target object description image, the to-be-replaced object description image, and the to-be-processed image are all used to describe the same object;

[0288] The updating unit is specifically used to update the image generation model according to the difference representation data between the target image and the image to be processed.

[0289] Based on the relevant contents of the above-mentioned image generating device 600, it can be known that for the image generating device 600 provided in the embodiment of the present application, firstly obtain a target object description image (for example, image data input by a user for describing the target object), an image describing an object to be replaced (for example, a video frame in the video data that needs to be replaced and can better represent the characteristics of the object to be replaced), key point representation data of the image describing the object to be replaced, and key point representation data of the image to be processed (for example, any video frame in the video data); then perform information fusion processing on the target object description image, the image describing the object to be replaced, the key point representation data of the image describing the object to be replaced, and the key point representation data of the image to be processed to obtain an image generating device 600. Conditional characterization data is generated so that the image generation conditional characterization data can characterize the information required for reference when performing image generation processing (for example, the characteristics of the target object itself, the state of the object to be replaced in the image to be processed, etc.); then, image generation processing is performed according to the image generation conditional characterization data to obtain a target image corresponding to the image to be processed, so that the target image and the target object description image are used to describe the same object, and the target image and the image to be processed are consistent in at least one object state characterization information (for example, object posture information, etc.), so that the target image can represent the object replacement result for the image to be processed, thereby achieving the purpose of performing object replacement processing on an existing image or an existing video. Among them, because the image generation condition representation data is determined based on multiple information (for example, the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed), the image generation condition representation data can more accurately and comprehensively represent the information required for reference when performing image generation processing, thereby effectively avoiding defects caused by insufficient or inaccurate reference information when performing image generation processing, and thus helping to improve the object replacement effect.

[0290] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the image generation method provided in the embodiment of the present application.

[0291] See also Figure 7, which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0292] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0293] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0294] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0295] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0296] The embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the image generation method provided in the embodiment of the present application.

[0297] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0298] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0299] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0300] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0301] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0302] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0303] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0304] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0305] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0306] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0307] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0308] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0309] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0310] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of the to-be-processed image; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object; Performing information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data; Image generation processing is performed according to the image generation condition representation data to obtain a target image corresponding to the image to be processed; the target image and the target object description image are both used to describe the same object, and the target image and the image to be processed are consistent in at least one object state representation information.

2. The method according to claim 1, characterized in that The process of determining the image generation condition characterization data includes: Performing feature extraction processing on the target object description image to obtain image representation features; Performing feature extraction processing on the to-be-replaced object description image and key point representation data of the to-be-replaced object description image to obtain multi-information representation features; Performing feature extraction processing on the key point representation data of the image to be processed to obtain key point representation features; The image representation features, the multi-information representation features and the key point representation features are subjected to cross-attention processing to obtain the image generation condition representation data; the key point representation features are used as query features in the cross-attention processing; the multi-information representation features are used as key features in the cross-attention processing; and the image representation features are used as value features in the cross-attention processing.

3. The method according to claim 2, characterized in that The cross attention processing comprises: Performing a dot product process on the key point representation feature and the multi-information representation feature to obtain a first dot product result; Performing a dot product process on the first dot product result and the image representation feature to obtain a second dot product result; The image generation condition representation data is determined based on the second dot product result and the information supplement feature; the information supplement feature is determined based on the target object description image, the multi-information representation feature, the key point representation feature, and the first dot product result.

4. The method according to claim 3, characterized in that The process of determining the information supplementary features includes: Splicing the target object description image with the multi-information representation feature to obtain a first splicing feature; Performing a dot product process on the first concatenated feature and the first dot product result to obtain a third dot product result; Concatenate the third dot product result with the key point representation feature to obtain a second concatenated feature; The information supplement feature is determined according to the second splicing feature.

5. The method according to claim 4, characterized in that The step of performing splicing processing on the target object description image and the multi-information representation feature to obtain a first splicing feature includes: The target object description image, the key point representation data of the target object description image and the multi-information representation feature are spliced ​​to obtain a first splicing feature.

6. The method according to claim 1, characterized in that The target image corresponding to the image to be processed is determined based on the image generation condition characterization data and the noise data corresponding to the image to be processed; If the image generation method is used to train some or all modules in an image generation model, the noise data is obtained by performing noise processing on the image to be processed; or, If the image generation method is used to perform object replacement processing on one or more images to be processed according to the target object description image, the noise data is randomly generated noise data.

7. The method according to claim 6, characterized in that The image generation method is used to perform object replacement processing on the video to be processed according to the target object description image; The image to be processed is used to represent any video frame in the video to be processed; Each video frame in the video to be processed corresponds to the same noise data.

8. The method according to claim 1, characterized in that The image generation method is used to perform object replacement processing on a video to be processed according to the target object description image; the video to be processed is determined according to a video specified operation; The target object description image is determined according to the image specifying operation; The object description image to be replaced refers to a video frame existing in the video to be processed and achieving the object representation performance target; The image to be processed is used to represent any video frame in the video to be processed.

9. The method according to claim 8, characterized in that The method further comprises: After obtaining the target images corresponding to at least two video frames in the video to be processed, the target images corresponding to the at least two video frames are arranged and processed according to the arrangement sequence numbers corresponding to the at least two video frames in the video to be processed to obtain the target video corresponding to the video to be processed.

10. The method according to claim 1, characterized in that The image generation method is used to train part or all of the modules in the image generation model according to at least one image sample group; For any one of the image sample groups, each image in the image sample group is used to describe the same object; The target object description image, the to-be-replaced object description image, and the to-be-processed image are determined from the at least one image sample group.

11. The method according to claim 10, characterized in that The target object description image, the to-be-replaced object description image, and the to-be-processed image are all determined from the same image sample group; or, The target object description image is determined from a first image sample group, the to-be-replaced object description image and the to-be-processed image are both determined from a second image sample group, the object described by the first image sample group is different from the object described by the second image sample group, and the at least one image sample group includes the first image sample group and the second image sample group.

12. The method according to claim 11, characterized in that If the target object description image, the to-be-replaced object description image, and the to-be-processed image are all determined from the same image sample group, the to-be-replaced object description image is determined based on the target object description image.

13. The method according to claim 1, characterized in that The step of fusing the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data includes: Using a conditional representation module in an image generation model, the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image are processed to obtain the image generation conditional representation data; The performing image generation processing according to the image generation condition representation data to obtain a target image corresponding to the image to be processed includes: The image generation module in the image generation model is used to process the image generation condition representation data to obtain a target image corresponding to the image to be processed.

14. The method according to claim 13, characterized in that After the image generation module in the image generation model is used to process the image generation condition representation data to obtain a target image corresponding to the image to be processed, the method further includes: Based on the difference representation data between the target image and the label image corresponding to the image to be processed, the image generation model is updated, and the step of using the conditional representation module in the image generation model to process the target object description image, the object description image to be replaced, the key point representation data of the object description image to be replaced, and the key point representation data of the image to be processed is returned to continue executing until a preset stop condition is reached.

15. The method according to claim 14, characterized in that The updating of the image generation model according to the difference representation data between the target image and the label image corresponding to the image to be processed includes: Based on the difference representation data between the target image and the label image corresponding to the image to be processed, other modules in the image generation model except the pre-trained module are updated; the pre-trained module is used to represent the pre-trained modules existing in the image generation model; the pre-trained module is the conditional representation module or the image generation module.

16. The method according to claim 14, characterized in that The target object description image, the to-be-replaced object description image and the to-be-processed image are all used to describe the same object; The updating of the image generation model according to the difference representation data between the target image and the label image corresponding to the image to be processed includes: The image generation model is updated according to the difference representation data between the target image and the image to be processed.

17. The method according to claim 1, characterized in that The key point representation data of the to-be-replaced object description image is used to describe the pixel distribution state in the to-be-replaced object description image; The key point representation data of the image to be processed is used to describe the pixel distribution state in the image to be processed.

18. An image generating device, characterized in that: include: an acquisition unit, configured to acquire a target object description image, a to-be-replaced object description image, key point representation data of the to-be-replaced object description image, and key point representation data of the to-be-processed image; the to-be-replaced object description image and the to-be-processed image are both used to describe the same object; a fusion unit, configured to perform information fusion processing on the target object description image, the to-be-replaced object description image, the key point representation data of the to-be-replaced object description image, and the key point representation data of the to-be-processed image to obtain image generation condition representation data; A generating unit, configured to perform image generation processing according to the image generation condition representation data to obtain a target image corresponding to the image to be processed; The target image and the target object description image are both used to describe the same object, and the target image and the image to be processed are consistent in at least one object state representation information.

19. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 17.

20. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 17.

Images