Training method of image editing model and image editing method and device

Abstract

The invention discloses a training method of an image editing model and an image editing method and device, and relates to the technical field of computers, in particular to the field of artificial intelligence such as computer vision and deep learning. According to the specific implementation scheme, an initial image editing model is adopted, and an intermediate feature vector is generated based on a sample image, a target scene image and a condition text; determining a joint attention map of the sample image and the target scene image according to the intermediate feature vector; determining a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image according to the joint attention map; and according to the difference between the first cross-view attention map and a first mask image corresponding to the sample image and the difference between the second cross-view attention map and a second mask image corresponding to the target scene image, training the initial image editing model to obtain a trained image editing model.

Description

Training methods for image editing models, image editing methods and devices Technical Field

[0001] This application relates to the field of computer technology, and in particular to the fields of artificial intelligence such as computer vision and deep learning, specifically to a training method for an image editing model, an image editing method, and an apparatus. Background Technology

[0002] In the field of artificial intelligence, image editing technology has undergone a significant transformation from simple pixel-level operations to complex, advanced processing methods based on deep learning. With the continuous development of AI technology, image editing is widely used in scenarios such as image completion, content deletion, and photo restoration, offering advantages such as high editing efficiency and a high degree of automation. Summary of the Invention

[0003] This application provides a training method for an image editing model, an image editing method, and an apparatus. The specific scheme is as follows: According to one aspect of this application, a training method for an image editing model is provided, comprising: using an initial image editing model, generating an intermediate feature vector based on a sample image, a target scene image, and conditional text; wherein, the target scene image and the sample image containing the target entity are images from different perspectives within the same scene, and the conditional text is used to describe the geometric relationship of the target entity in the target scene image; determining a joint attention map of the sample image and the target scene image based on the intermediate feature vector; determining a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image based on the joint attention map; training the initial image editing model based on the difference between the first cross-view attention map and the first mask image corresponding to the sample image, and the difference between the second cross-view attention map and the second mask image corresponding to the target scene image, to obtain a trained image editing model; wherein, the first mask image and the second mask image are used to quantify the geometric consistency between the sample image and the target scene image.

[0004] According to another aspect of this application, an image editing method is provided, comprising: acquiring a reference scene image and conditional text; wherein the conditional text is used to describe the geometric relationship of the entity to be generated in the reference scene image; and generating a target image containing the entity to be generated by employing an image editing model based on the reference scene image and the conditional text; wherein the image editing model is trained using the method of the above-described embodiment.

[0005] According to another aspect of this application, a training apparatus for an image editing model is provided, comprising: a first generation module, configured to generate an intermediate feature vector based on a sample image, a target scene image, and conditional text using an initial image editing model; wherein the target scene image and the sample image containing the target entity are images from different perspectives within the same scene, and the conditional text is used to describe the geometric relationship of the target entity in the target scene image; a first determination module, configured to determine a joint attention map of the sample image and the target scene image based on the intermediate feature vector; a second determination module, configured to determine a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image based on the joint attention map; and a training module, configured to train the initial image editing model based on the difference between the first cross-view attention map and a first mask image corresponding to the sample image, and the difference between the second cross-view attention map and a second mask image corresponding to the target scene image, to obtain a trained image editing model; wherein the first mask image and the second mask image are used to quantify the geometric consistency between the sample image and the target scene image.

[0006] According to another aspect of this application, an image editing apparatus is provided, comprising: an acquisition module for acquiring a reference scene image and conditional text; wherein the conditional text is used to describe the geometric relationship of an entity to be generated in the reference scene image; and a generation module for generating a target image containing the entity to be generated by using an image editing model based on the reference scene image and the conditional text; wherein the image editing model is trained using the method of the above-described embodiment.

[0007] According to another aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.

[0008] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0009] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation thereof. Specifically: Figure 1 is a flowchart illustrating a training method for an image editing model according to an embodiment of the present invention; Figure 2 is a flowchart illustrating a training method for an image editing model according to another embodiment of the present invention; Figure 3 is a flowchart illustrating a training method for an image editing model according to another embodiment of the present invention; Figure 4 is a flowchart illustrating an image editing method according to an embodiment of the present invention; Figure 5 is a schematic diagram illustrating the training process of an image editing model according to an embodiment of the present invention; Figure 6 is a schematic diagram illustrating the structure of a training device for an image editing model according to an embodiment of the present invention; Figure 7 is a schematic diagram illustrating the structure of an image editing device according to an embodiment of the present invention; Figure 8 is a block diagram of an electronic device used to implement the training method for the image editing model according to an embodiment of the present invention. Detailed Implementation

[0012] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that the acquisition, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0014] The following description, with reference to the accompanying drawings, describes a training method, image editing method, apparatus, electronic device, and storage medium for an image editing model according to embodiments of this application.

[0015] Figure 1 is a flowchart illustrating the training method of an image editing model provided in an embodiment of this application.

[0016] The training method of the image editing model in this application embodiment can be executed by the image editing model training device in this application embodiment, which can be configured in an electronic device.

[0017] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0018] As shown in Figure 1, the training method of the image editing model includes: Step 101, using an initial image editing model, generating an intermediate feature vector based on the sample image, the target scene image and the conditional text.

[0019] In this application, the sample image contains the target entity, while the target scene image does not contain the target entity. The target scene image and the sample image are images from different perspectives within the same scene.

[0020] For example, the target entity can be a person, an object, or an animal, and the number of target objects can be one or more, without limitation.

[0021] For example, the target scene image can be obtained by image processing of the sample image, or it can be generated using a large model, without limitation.

[0022] In this application, conditional text can be used to describe the geometric relationship of the target entity to be generated in the target scene image.

[0023] For example, geometric relationships may include, but are not limited to, the position, orientation, relative size, and spatial relationship with other entities of the target entity to be generated in the target scene image.

[0024] For example, the conditional text is "A wooden bookshelf filled with books and decorative items is located to the left of the sofa".

[0025] For example, conditional text can also be used to describe the semantic attributes of a target entity in a target scene image, such as category, appearance, and state. Thus, conditional text can be used to ensure that the generated image visually conforms to the physical constraints of the real world.

[0026] For example, the condition text is "A lady wearing a red dress is standing in front of a white background".

[0027] For example, an image encoder, such as a VAE (Variational Autoencoder), can be used to encode the sample image and the target scene sample image respectively to obtain the first image latent variable and the second image latent variable. A text encoder can be used to encode the conditional text to obtain the text latent variable. The first image latent variable is then noise-added to obtain the noisy latent variable. The noisy latent variable, the second image latent variable, and the text latent variable are then fused to obtain the fused latent variable. Finally, an initial image editing model is used to extract intermediate features from the fused latent variable to obtain the intermediate feature vector.

[0028] For example, the initial image editing model may include multiple network layers. The latent variables are fused and passed through multiple network layers for feature extraction. The current network layer can extract features from the feature vector output by the previous network layer to obtain an intermediate feature vector.

[0029] For example, the initial image editing model can be a text-driven image editing diffusion model based on the MMDIT (Multimodal Diffusion Transformer) architecture, or a diffusion model based on the U-Net architecture, or an editing model with other multimodal Transformer structures, without limitation.

[0030] Step 102: Determine the joint attention map of the sample image and the target scene image based on the intermediate feature vector.

[0031] In this application, the joint attention map can reflect the structural correspondence between the sample image and the target scene image.

[0032] For example, a joint attention map can be obtained by using self-attention computation based on the intermediate feature vector.

[0033] Step 103: Based on the joint attention map, determine the first cross-view attention map of the sample image and the second cross-view attention map of the target scene image.

[0034] In this application, the first cross-view attention map can be used to characterize the structural correspondence between each position in the sample image and the target scene image, reflect the structural interaction strength between the sample image and the target scene image, and can be used to improve the scene consistency of the generated image.

[0035] Similarly, the second cross-view attention map can be used to characterize the structural correspondence between each position in the target scene image and the sample image, reflect the structural interaction strength between the target scene image and the sample image, and can be used to improve the scene consistency of the generated image.

[0036] For example, a first cross-view attention map and a second cross-view attention map can be determined based on the attention intensity of each position in the sample image to each position in the target scene image and the attention intensity of each position in the target scene image to each position in the sample image in the joint attention map.

[0037] For example, the first cross-view attention map can be expressed as a first cross-view attention map, and the second cross-view attention map can be expressed as a second cross-view attention map. This application does not limit the names of the first cross-view attention map and the second cross-view attention map.

[0038] Step 104: Based on the difference between the first cross-view attention map and the first mask image corresponding to the sample image, and the difference between the second cross-view attention map and the second mask image corresponding to the target scene image, train the initial image editing model to obtain the trained image editing model.

[0039] In this application, the first mask image and the second mask image can be used to quantify the geometric consistency between the sample image and the target scene image.

[0040] For example, regions in the first mask image and the second mask image with values ​​greater than the first threshold can represent geometrically corresponding scene regions in the sample image and the target scene image, and can be used to guide the distribution of attention.

[0041] For example, the first mask image and the second mask image can be obtained based on the sparse keypoint matching relationship between the sample image and the target scene image. For example, the sparse keypoint matching relationship can be used to describe which location point in the sample image matches which location point in the target scene image.

[0042] In this application, the model loss can be determined based on the difference between the first cross-view attention map and the first mask image, and the difference between the second cross-view attention map and the second mask image. Based on the model loss, the parameters of the initial image editing model are adjusted, and the model with adjusted parameters is trained until the training termination condition is met.

[0043] For example, the training termination condition could be that the model loss is less than the loss threshold, or that the maximum number of training iterations has been reached, etc., and there are no restrictions on this.

[0044] Since the regions with larger values ​​in the first and second mask images are geometrically corresponding scene regions in the sample image and the target scene image, training the initial image editing model based on the differences between the first cross-view attention map and the first mask image, as well as the differences between the second cross-view attention map and the second mask image, can encourage the model to give higher weights to the positions of regions with larger values ​​in the mask images during cross-view attention. This can explicitly guide the model to pay attention to the background regions that are geometrically corresponding across views during the editing process, thereby improving scene consistency across viewpoints.

[0045] The image editing model of this application embodiment can be applied to scenarios where a target image is generated given a reference scene image and conditional text.

[0046] For example, the image editing model of this application embodiment can be used in scenarios such as multi-camera consistent image generation, continuous comic or storyboard generation, AR (Augmented Reality) or VR (Virtual Reality) scene content generation, and video first and last frame generation.

[0047] In this embodiment, an intermediate feature vector is generated based on the sample image, the target scene image, and the conditional text. Based on this vector, a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image are further determined. This can improve the model's understanding and processing capabilities for cross-view scenes. The initial image editing model is trained using the differences between the first cross-view attention map and the first mask image, as well as the differences between the second cross-view attention map and the second mask image. This can constrain the consistency between the cross-view attention map and the geometrically corresponding mask, enabling the model to learn how to establish high-attention connections between geometrically corresponding regions. This not only strengthens the model's understanding of the scene's geometric structure but also allows the model to more accurately locate and apply these geometric relationships in new scenes. When executing conditional text, the model tends to edit in geometrically reasonable background areas, avoiding damage to the entire scene. At the same time, it takes into account both text conformity and scene geometric consistency, improving the quality and realism of cross-view image editing.

[0048] Figure 2 is a flowchart illustrating the training method of an image editing model provided in another embodiment of this application.

[0049] As shown in Figure 2, the training device for the image editing model includes: Step 201, using an initial image editing model to generate an intermediate feature vector based on the sample image, the target scene image, and the conditional text.

[0050] In this application, step 201 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0051] Step 202: Extract the first visual feature vector of the sample image and the second visual feature vector of the target scene image from the intermediate feature vector.

[0052] In this application, since the intermediate feature vector may include the visual feature vector of the sample image, the visual feature vector of the target scene image, and the text feature vector of the conditional text, the first visual feature vector of the sample image and the second visual feature vector of the target scene image can be extracted from the intermediate feature vector.

[0053] Step 203: Determine the joint attention map based on the first visual feature vector and the second visual feature vector.

[0054] In some embodiments, the first visual feature vector and the second visual feature vector can be concatenated to obtain a concatenated feature vector, and self-attention enhancement can be applied to the concatenated feature vector to obtain a joint attention map.

[0055] For example, the first visual feature vector is The size is The second visual feature vector is The size is Then the concatenated feature vector is The size is During self-attention calculation, the query matrix AND key matrix Both are concatenated feature vectors , that is The joint attention map can be calculated using the following formula. :

[0056] Among them, joint attention map The size is .

[0057] For example, one or more intermediate attention layers in the initial image acquisition model can be used to compute a joint attention map using the method described above.

[0058] Therefore, by concatenating the first visual feature vector and the second visual feature vector, and performing self-attention calculation on the concatenated vector, a joint attention map of the sample image and the target scene image is obtained. The calculation method is simple and convenient, and can improve computational efficiency.

[0059] Step 204: Based on the joint attention map, determine the first cross-view attention map of the sample image and the second cross-view attention map of the target scene image.

[0060] In this application, step 204 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0061] In some embodiments, the first visual feature vector can be divided into multiple first visual sub-vectors, and the second visual feature vector can be divided into multiple second visual sub-vectors. That is, the first visual feature vector includes multiple first visual sub-vectors, and the second visual feature vector includes multiple second visual sub-vectors. Here, the visual sub-vectors can be understood as basic unit representations obtained by feature extraction from the image, and visual sub-vectors are abstract representations of local information in the image.

[0062] For example, the first cross-view attention map can be determined by the following method: For any first visual sub-vector, the first attention intensity of the first visual sub-vector to the second visual sub-vector and the second attention intensity of the second visual sub-vector to the first visual sub-vector can be determined according to the indices of the first visual sub-vector and the second visual sub-vector in the concatenated feature vector, respectively. Then, the cross-view attention intensity between the first visual sub-vector and the target scene image can be determined according to the first attention intensity and the second attention intensity. Finally, the cross-view attention intensity corresponding to each first visual sub-vector can be sorted according to the spatial position information of the first visual sub-vector to obtain the first cross-view attention map.

[0063] For example, the cross-view attention intensity corresponding to all first visual sub-vectors can be rearranged according to spatial location information to obtain the first cross-view attention map.

[0064] Therefore, the attention intensity of the first visual sub-vector to the second visual sub-vector and the attention intensity of the second visual sub-vector to the first visual sub-vector can be determined from the joint attention map. Based on these two attention intensities, the cross-view attention map of the sample image can be determined, which can make the cross-view attention map contain the interaction intensity between each point in the sample image and each point in the target scene image, with high accuracy.

[0065] For example, the cross-view attention intensity between the first visual sub-vector and the target scene image can be determined by the following method: For any first visual sub-vector, the average of the first attention intensity of the first visual sub-vector to all second visual sub-vectors can be calculated to obtain the first average intensity; the average of the second attention intensity of all second visual sub-vectors to the first visual sub-vector can be calculated to obtain the second average intensity; and the average of the first average intensity and the second average intensity can be calculated to obtain the cross-view attention intensity between the first visual sub-vector and the target scene image.

[0066] For example, cross-view attention intensity can be used to reflect the interaction between regions in a sample image and their geometrically corresponding regions in a target scene image.

[0067] As an example, based on the above example, the index set of the first visual sub-vector can be denoted as... Let the set of indices of the second visual subvector be denoted as . The cross-view attention intensity between any first visual sub-vector token p in the sample image and the target scene image can be calculated using the following formula:

[0068] Among them, the first item This represents the average attention intensity of token p over all second visual subvectors; that is, the first term is the first average intensity, and the second term... This represents the average attention intensity of all second visual subvectors on token p, i.e., the second term is the second average intensity. The average of the two terms yields the cross-view attention intensity between token p and the target scene image. , or what is called symmetrical cross-view interaction intensity; This represents the attention intensity of token p on the second visual subvector token q. It can be obtained by querying the joint attention map based on the indices of token p and token q in the concatenated feature vector; This represents the attention intensity of token q towards token p. It can be obtained by querying the joint attention map based on the indices of token q and tokenp in the concatenated feature vector.

[0069] For example, suppose , The dimension of the joint attention map is , that is The index of token p The index of token q So from the joint attention map Get from The value of this value represents the attention strength of token p to token q, while the joint attention graph... middle The value is the index. The token q is indexed as The attention intensity of token p.

[0070] Therefore, based on the average attention intensity of the first visual sub-vector to all second visual sub-vectors, and the average attention intensity of all second visual sub-vectors to the first visual sub-vector, the cross-view attention intensity is determined. This takes into account both the average attention intensity of a region in the sample image to all regions in the target scene image, and the average attention intensity of each region in the target scene image to that region in the sample image. This allows the cross-view attention intensity to accurately reflect the interaction between a region in the sample image and the geometrically corresponding region in the target scene image.

[0071] In some embodiments, a second cross-view attention map may be determined using a method similar to that used to determine the first cross-view attention map.

[0072] For example, the second cross-view attention map can be determined by the following method: For any second visual sub-vector, the cross-view attention intensity between the second visual sub-vector and the sample image can be determined based on the first attention intensity and the second attention intensity. Then, based on the spatial position information of the second visual sub-vector, the cross-view attention intensity corresponding to each second visual sub-vector is sorted to obtain the second cross-view attention map.

[0073] For example, the cross-view attention intensity corresponding to all second visual sub-vectors can be rearranged according to spatial location information to obtain the second cross-view attention map.

[0074] For example, the cross-view attention intensity of the second visual sub-vector and the sample image can be determined by the following method: the average of the second attention intensity of the second visual sub-vector to all first visual sub-vectors can be calculated to obtain the third average intensity; the average of the first attention intensity of all first visual sub-vectors to the second visual sub-vector can be calculated to obtain the fourth average intensity; and the average of the third average intensity and the fourth average intensity can be calculated to obtain the cross-view attention intensity of the second visual sub-vector and the sample image.

[0075] As an example, based on the above example, the cross-view attention intensity between any second visual sub-vector token q in the target scene image and the sample image can be calculated using the following formula:

[0076] Among them, the first item This represents the average attention intensity of token q across all first visual subvectors; that is, the first term is the third average intensity, and the second term... This represents the average attention intensity of all first visual subvectors on token q, which is also the fourth average intensity.

[0077] Step 205: Based on the difference between the first cross-view attention map and the first mask image corresponding to the sample image, and the difference between the second cross-view attention map and the second mask image corresponding to the target scene image, the initial image editing model is trained to obtain the trained image editing model.

[0078] In this application, step 205 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0079] In this embodiment, by generating a joint attention map using a first visual feature vector and a second visual feature vector, the geometric relationship and spatial transformation information between the sample image and the target scene image can be effectively captured. This helps maintain the consistency of image content from different perspectives and enhances cross-perspective consistency. Furthermore, by generating the joint attention map using the first and second visual feature vectors, not only are the features within a single image considered, but the feature correspondences from different perspectives are also integrated. Therefore, it can more accurately identify and retain key details and structural information in the image, improving detail retention and resulting in more natural and realistic image editing results.

[0080] Figure 3 is a flowchart illustrating the training method of an image editing model provided in another embodiment of this application.

[0081] As shown in Figure 3, the training method of the image editing model includes: Step 301, using the initial image editing model, generating intermediate feature vectors based on sample images, target scene images and conditional text.

[0082] In this application, step 301 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0083] In some embodiments, the target scene image can be obtained based on the sample image in the following manner: the target entity in the sample image is removed to obtain an initial scene image; based on the initial scene image, a multi-view image sequence under the same scene is generated; sparse keypoint matching is performed between the candidate scene images in the multi-view image sequence and the sample image to obtain matching information; and then the target scene image is determined from the candidate scene images based on the matching information.

[0084] For example, a pre-trained entity segmentation model can be used to locate the target entity region in a sample image, and the target entity region can be input into an image inpainting model to perform a target entity removal operation, generating an initial scene image containing only the background. The initial scene image can serve as the basis for generating scene images from multiple perspectives in the future.

[0085] Optionally, to improve the quality of the initial scene image, if artifacts or broken background structures exist in the initial scene image, it can be discarded and a new initial scene image can be generated.

[0086] For example, for sample images that mainly consist of human figures, a human body segmentation model can be used to locate the target human figure region, and the target human figure region can be input into an image restoration model to perform a human figure removal operation.

[0087] For example, if the target entity is a non-human figure such as an object, the sample image and the initial scene image can be obtained directly using an existing object removal dataset.

[0088] For example, a multi-view image sequence may include candidate scene images with different perspectives than the sample images. For example, a multi-view image sequence may be generated based on an initial scene image using an image-to-video diffusion model.

[0089] For example, the image-to-video diffusion model is trained on large-scale natural videos and automatically simulates camera movements such as translation, rotation, and zoom, thereby generating multiple frames of the same physical scene from different perspectives or compositions.

[0090] It should be noted that a multi-view image sequence may include one or more candidate scene images, and there is no limitation on this.

[0091] For example, the matching information may include sparse keypoint matching relationships, and may include the matching relationship between specific location points in the sample image and corresponding location points in the candidate scene image, the matching degree between the sample image and the candidate scene image, etc.

[0092] For example, to ensure that two images belong to the same scene, a feature matching network can be used to perform sparse keypoint matching between the sample image and the candidate scene image. For instance, ,in, and These represent the nth image in the sample image and the candidate scene image, respectively. The coordinates of the matching points, that is, the location points in the sample image. Location points in candidate scene images match.

[0093] For example, an optical flow estimation network can also be used to perform sparse keypoint matching between the sample image and the candidate scene image.

[0094] For example, the target scene image can be determined from the candidate scene images in the following way: the number of matching points between the candidate scene image and the target scene image can be determined based on the matching information, and the target scene image can be determined from the candidate scene images based on the number of matching points.

[0095] As an example, a candidate scene image with more than a first preset number of matching location points can be used as the target scene image. For instance, if the preset number is 10, and a candidate scene image matches 15 location points with a sample image (meaning there are 15 pairs of matching location points between the two images), exceeding the preset number, then that candidate scene image can be used as the target scene image.

[0096] As another example, candidate scene images can be sorted in descending order of the number of matched location points, and the second-most preset number of candidate scene images in the sorted results can be used as the target scene image.

[0097] Therefore, based on the number of matching points between candidate scene images and sample images, high-quality target scene images that satisfy the same scene but have different perspectives can be selected from the candidate scene images. Training the model based on high-quality target scene images can improve the accuracy of the images generated by the model.

[0098] In this embodiment, scene images from different perspectives are generated based on a single sample image, and target scene images are selected based on sparse keypoint matching results, which can improve the efficiency of training data construction. Furthermore, by quantifying the geometric consistency between candidate scene images and sample images through sparse keypoint matching, images with matching perspectives, structures, and spatial layouts can be accurately identified from multi-view image sequences as target scene images, thus improving the accuracy of target scene image selection.

[0099] In addition, since the target scene image originates from the same initial scene as the sample image and has been geometrically aligned, its background content is highly compatible with the sample image, which helps to maintain a natural transition of visual context and semantic coherence during the editing process.

[0100] Step 302: Determine the joint attention map of the sample image and the target scene image based on the intermediate feature vector.

[0101] Step 303: Based on the joint attention map, determine the first cross-view attention map of the sample image and the second cross-view attention map of the target scene image.

[0102] In this application, steps 302-303 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0103] Step 304: Determine the geometrically guided attention loss based on the difference between the first cross-view attention map and the first mask image corresponding to the sample image, and the difference between the second cross-view attention map and the second mask image corresponding to the target scene image.

[0104] In some embodiments, a first position point in the sample image is matched with a second position point in the target scene image. The first mask image and the second mask image can be obtained by the following method: an initialized third mask image and a fourth mask image can be obtained, both of which are all-zero mask images. The center point of the target radial kernel is translated and aligned to the first position point in the third mask image, and the elements of the target radial kernel are added to the corresponding elements of the third mask image to obtain the first mask image. The center point of the target radial kernel is translated and aligned to the second position point in the fourth mask image, and the elements of the target radial kernel are added to the corresponding elements of the fourth mask image to obtain the second mask image.

[0105] For example, the size of the third mask image is the same as the size of the sample image, and the size of the fourth mask image is the same as the size of the target scene image.

[0106] For example, the target radial kernel is a kernel function in which the value decreases as the distance from the center point increases. For example, the target radial kernel can be a Gaussian kernel, which is a convolution kernel based on a Gaussian function (normal distribution function). Each element value in the Gaussian kernel is calculated using the Gaussian function, and its value is largest at the center of the kernel, gradually decreasing towards the edges, exhibiting a bell-shaped distribution.

[0107] For example, one The Gaussian kernel can be represented as follows:

[0108] The core has the largest element value at its center and smaller element values ​​around it.

[0109] For example, since the element values ​​in the mask image obtained by aligning and stacking the target radial kernel may be greater than a preset value, such as 1, the obtained mask image can be normalized so that its values ​​fall within a certain range.

[0110] For example, if there are multiple pairs of matching first and second position points between the sample image and the target scene image, for each pair of matching position points, the target radial kernel can be translated in space so that the center point of the target radial kernel is aligned with the first position point. The element value of the target radial kernel is then added to the element at the corresponding position in the third mask image. The above operation is repeated for all first position points in the third mask image to complete the multi-kernel superposition operation and obtain the fifth mask image. The target radial kernel is then translated in space so that the center point of the target radial kernel is aligned with the second position point. The element value of the target radial kernel is then added to the element at the corresponding position in the fourth mask image. The above operation is repeated for all second position points in the fourth mask image to complete the multi-kernel superposition operation and obtain the sixth mask image.

[0111] Next, the fifth and sixth mask images are normalized and cropped so that their values ​​fall within a certain range to obtain the seventh and eighth mask images. Then, the seventh and eighth mask images are downsampled to the same resolution as the intermediate feature vector to obtain the first and second mask images.

[0112] It should be noted that, in this application, the first mask image and the second mask image can be obtained in advance before training, or they can be obtained during training; there is no limitation on this.

[0113] Therefore, based on the matching position points between the sample image and the target scene image, the target radial kernel is aligned and superimposed with the matching position points in the initialized third and fourth mask images to obtain the mask images corresponding to the sample image and the target scene image. This distinguishes the geometrically corresponding scene regions in the sample image and the target scene image from other non-corresponding scene regions, and converts the matching position points into soft geometric correspondence masks suitable for attention supervision to guide the attention distribution.

[0114] In some embodiments, a first attention loss may be determined based on the difference between a first cross-view attention map and a first mask image, a second attention loss may be determined based on the difference between a second cross-view attention map and a second mask image, and a geometry-guided attention loss may be determined based on the first attention loss and the second attention loss.

[0115] For example, the mean square error between the first cross-view attention map and the first mask image can be calculated to obtain the first attention loss, the mean square error between the second cross-view attention map and the second mask image can be calculated to obtain the second attention loss, and the first attention loss and the second attention loss can be added together to obtain the geometrically guided attention loss.

[0116] For example, the geometrically guided attention loss can be determined using the following formula:

[0117]

[0118]

[0119] in, It is the first attention loss, It is the second attention loss. It is geometrically guided attention loss. It refers to the number of spatial locations. and These are the first cross-view attention map and the second cross-view attention map. and These are the first mask image and the second mask image, respectively. It is the L2 norm squaring operation.

[0120] For example, the first attention loss and the second attention loss can also use KL divergence, cross-entropy, contrastive loss, etc., without limitation.

[0121] Step 305: Train the initial image editing model using geometrically guided attention loss to obtain the trained image editing model.

[0122] In this application, the noise prediction loss can be determined based on the difference between the predicted noise and the actual noise of the initial image editing model. The initial image editing model can be trained based on the geometrically guided attention loss and the noise prediction loss to obtain the trained image editing model.

[0123] For example, the mean square error between the predicted noise and the actual noise can be calculated and used as the noise prediction loss.

[0124] For example, the geometrically guided attention loss and the noise prediction loss can be weighted to obtain the model loss. Based on the model loss, the parameters of the initial image editing model can be adjusted until the training termination condition is met, resulting in a trained image editing model.

[0125] As an example, the model loss can be obtained using the following formula. :

[0126] in, It is the noise prediction loss. These are the weights of the geometry-guided attention loss. For example, a geometry-guided attention loss can be applied to several attention layers in the middle of the image editing model to balance geometric constraints with generative diversity.

[0127] Therefore, by training the initial image editing model using geometrically guided attention loss combined with noise prediction loss, the accuracy of the generated images and the geometric consistency of the scene can be improved.

[0128] In this embodiment, a first attention loss is determined based on the difference between the first cross-view attention map and the first mask image, and a second attention loss is determined based on the difference between the second cross-view attention map and the second mask image. Based on these two attention losses, a geometrically guided attention loss is obtained. The initial image editing model is trained based on this geometrically guided attention loss, which can display the background regions of the cross-view geometrically corresponding sample images and target scene images that the guided model focuses on, thereby improving cross-view scene consistency.

[0129] To implement the above embodiments, this application also proposes an image editing method. Figure 4 is a schematic flowchart of an image editing method provided in an embodiment of this application.

[0130] As shown in Figure 4, the image editing method includes: Step 401, obtaining a reference scene image and conditional text.

[0131] For example, the conditional text can be used to describe the geometric relationship of the entity to be generated in a reference scene image. For instance, the entity to be generated can be a person, an object, an animal, etc., without limitation.

[0132] For example, the conditional text can also be used to describe the semantic attributes of the entity to be generated.

[0133] It should be noted that the number of entities to be generated can be one or more, and there is no limit to this.

[0134] Step 402: Based on the reference scene image and conditional text, use an image editing model to generate a target image containing the entity to be generated.

[0135] In this application, the image editing model can be trained using the training method of any of the above embodiments.

[0136] In this application, an image editing model can be used to edit a reference scene image based on conditional text to generate a target image containing the entity to be generated.

[0137] For example, an image encoder can be used to encode the reference scene image to obtain an image encoding vector of the reference scene image. A text encoder can be used to encode the conditional text to obtain a text encoding vector. The image encoding vector and the text encoding vector are then fused to obtain a fused encoding vector. The fused encoding vector is then input into an image editing model to obtain a target image containing the entity to be generated generated by the image editing model.

[0138] In this embodiment, by using the image editing model trained by the above training method based on the reference scene image and conditional text, a target image containing the entity to be generated is generated. This ensures that the target image not only meets the conditional text description but also has scene geometry consistent with the reference scene image, thereby improving the image generation quality.

[0139] The image editing model trained using the training method implemented in this application generates images that allow the generated target image to remain in the same physical environment as the reference scene image (such as the same street, room, building, landscape, etc.). Newly added entities (such as people, objects, etc.) have physically reasonable spatial or scale relationships in the scene, and the perspective and composition can be changed according to the text requirements while maintaining the consistency of the scene's geometric structure.

[0140] To facilitate understanding of the present application, the following description is provided in conjunction with Figure 5, which is a schematic diagram of the training process of an image editing model provided in an embodiment of the present application.

[0141] As shown in Figure 5, a VAE encoder 510 can be used to process the sample images. Scene sample images Encode them separately to obtain image latent variables Image latent variables The conditional text C is encoded using a text encoder 520 to obtain text latent variables, and the image latent variables are then processed. Noise is added to obtain noisy latent variables. The noisy latent variables and image latent variables are then combined. After text latent variable fusion, the data is input into MMDiT 530 (e.g., a network with 60 layers). In MMDiT 530, the intermediate attention layer (e.g., the 30th layer) calculates the joint attention map using the method described in the above embodiments. Then, based on the joint attention map, the cross-view attention map of the sample image is obtained. Cross-view attention map of target sample image .

[0142] Then, based on the cross-view attention map With mask image The differences between them are used to calculate the attention loss based on the cross-view attention map. With mask image The difference between the two is used to calculate the attention loss. Based on these two attention losses, the geometrically guided attention loss is obtained. Based on the geometrically guided attention loss and the noise prediction loss, the model loss is obtained. Based on the model loss, the initial image editing model is trained.

[0143] For example, the training method of the image editing model of this application can be deployed on a GPU cluster or cloud server for multi-GPU distributed training, and supports images with arbitrary aspect ratios through a multi-resolution training strategy, with the upper limit of the long side being, for example, 1024 pixels.

[0144] During the inference phase, the user provides reference scene images and text descriptions (e.g., "a girl is standing in front of a lighthouse, from a top-down perspective"). The image editing model can output multi-view generated images that conform to the text instructions and are geometrically consistent with the reference scene, and can return single-frame or multi-frame images as needed.

[0145] The solution of this application has the following technical effects: (1) It can significantly improve the geometric consistency of the scene. When generating cross-viewpoints and cross-compositions, it can maintain the consistency of the relative positions and proportions of scene elements such as buildings, furniture, roads, and horizons.

[0146] (2) Balancing text conformity and scene preservation: By using geometric guidance to reduce attention loss, the model tends to modify the background area with reasonable geometry when executing text instructions, thus avoiding destroying the entire scene.

[0147] (3) No explicit 3D reconstruction is required, resulting in low deployment costs. It does not rely on camera parameter estimation and 3D scene reconstruction, and can be trained and inferred directly at the 2D image level. It can achieve scene-consistent editing model with geometric perception capabilities by using only 2D images and image-to-video diffusion models without explicit 3D modeling or precise camera parameters. Furthermore, for existing image editing products, only a few adaptation modules and training processes need to be added to the original model to upgrade and obtain the "scene-consistent editing" capability.

[0148] To implement the above embodiments, this application also proposes a training device for an image editing model. Figure 6 is a schematic diagram of the structure of an image editing model training device provided in an embodiment of this application.

[0149] As shown in Figure 6, the training device 600 for the image editing model includes: a first generation module 610, used to generate an intermediate feature vector based on a sample image, a target scene image, and conditional text using an initial image editing model; wherein the target scene image and the sample image containing the target entity are images from different perspectives within the same scene, and the conditional text is used to describe the geometric relationship of the target entity in the target scene image; a first determination module 620, used to determine a joint attention map of the sample image and the target scene image based on the intermediate feature vector; a second determination module 630, used to determine a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image based on the joint attention map; and a training module 640, used to train the initial image editing model based on the difference between the first cross-view attention map and the first mask image corresponding to the sample image, and the difference between the second cross-view attention map and the second mask image corresponding to the target scene image, to obtain a trained image editing model; wherein the first mask image and the second mask image are used to quantify the geometric consistency between the sample image and the target scene image.

[0150] Optionally, the first determining module 620 is configured to: extract a first visual feature vector of the sample image and a second visual feature vector of the target scene image from the intermediate feature vector; and determine a joint attention map based on the first visual feature vector and the second visual feature vector.

[0151] Optionally, the first determining module 620 is used to: concatenate the first visual feature vector and the second visual feature vector to obtain a concatenated feature vector; and perform self-attention enhancement on the concatenated feature vector to obtain a joint attention map.

[0152] Optionally, the first visual feature vector includes a first visual sub-vector, and the second visual feature vector includes a second visual sub-vector. The second determining module 630 is used to: determine the first attention intensity of the first visual sub-vector to the second visual sub-vector and the second attention intensity of the second visual sub-vector to the first visual sub-vector based on the indices of the first visual sub-vector and the second visual sub-vector in the concatenated feature vector; determine the cross-view attention intensity of the first visual sub-vector and the target scene image based on the first attention intensity and the second attention intensity; and sort the cross-view attention intensity based on the spatial position information of the first visual sub-vector to obtain a first cross-view attention map.

[0153] Optionally, the second determining module 630 is configured to: obtain a first average intensity based on the average of the first attention intensity of the first visual sub-vector to all second visual sub-vectors; obtain a second average intensity based on the average of the second attention intensity of all second visual sub-vectors to the first visual sub-vectors; and determine the cross-view attention intensity based on the first average intensity and the second average intensity.

[0154] Optionally, a first position point in the sample image is matched with a second position point in the target scene image. The first and second mask images are obtained using the following modules: a first acquisition module, used to acquire an initialized third and fourth mask image; wherein the size of the third mask image is consistent with the size of the sample image, and the size of the fourth mask image is consistent with the size of the target scene image; a second acquisition module, used to translate and align the center point of the target radial kernel to the first position point in the third mask image, and add the elements of the target radial kernel to the corresponding elements of the third mask image to obtain the first mask image; a third acquisition module, used to translate and align the center point of the target radial kernel to the second position point in the fourth mask image, and add the elements of the target radial kernel to the corresponding elements of the fourth mask image to obtain the second mask image.

[0155] Optionally, the target scene image is obtained using the following modules: a fourth acquisition module, used to remove target entities from the sample image to obtain an initial scene image; a second generation module, used to generate a multi-view image sequence of the same scene based on the initial scene image; wherein the multi-view image sequence includes candidate scene images with different viewpoints from the sample image; a matching module, used to perform sparse keypoint matching between the candidate scene images and the sample image to obtain matching information; and a third determination module, used to determine the target scene image from the candidate scene images based on the matching information.

[0156] Optionally, the third determining module is used to: determine the number of location points that match between the candidate scene image and the sample image based on the matching information; and determine the target scene image from the candidate scene image based on the number of matching location points.

[0157] Optionally, the training module 640 is configured to: determine a first attention loss based on the difference between the first cross-view attention map and the first mask image; determine a second attention loss based on the difference between the second cross-view attention map and the second mask image; determine a geometrically guided attention loss based on the first attention loss and the second attention loss; and train the initial image editing model based on the geometrically guided attention loss to obtain a trained image editing model.

[0158] Optionally, the training module 640 is used to: determine the noise prediction loss based on the difference between the predicted noise and the real noise of the initial image editing model; and train the initial image editing model based on the geometrically guided attention loss and the noise prediction loss to obtain a trained image editing model.

[0159] It should be noted that the explanation of the aforementioned training method embodiment for the image editing model also applies to the training device for the image editing model in this embodiment, and therefore will not be repeated here.

[0160] In this embodiment, an intermediate feature vector is generated based on the sample image, the target scene image, and the conditional text. Based on this vector, a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image are further determined. This can improve the model's understanding and processing capabilities for cross-view scenes. The initial image editing model is trained using the differences between the first cross-view attention map and the first mask image, as well as the differences between the second cross-view attention map and the second mask image. This can constrain the consistency between the cross-view attention map and the geometrically corresponding mask, enabling the model to learn how to establish high-attention connections between geometrically corresponding regions. This not only strengthens the model's understanding of the scene's geometric structure but also allows the model to more accurately locate and apply these geometric relationships in new scenes. When executing conditional text, the model tends to edit in geometrically reasonable background areas, avoiding damage to the entire scene. At the same time, it takes into account both text conformity and scene geometric consistency, improving the quality and realism of cross-view image editing.

[0161] To implement the above embodiments, this application also proposes an image editing device. Figure 7 is a schematic diagram of the structure of an image editing device provided in an embodiment of this application.

[0162] As shown in Figure 7, the image editing device 700 includes: an acquisition module 710, used to acquire a reference scene image and conditional text; wherein the conditional text is used to describe the geometric relationship of the entity to be generated in the reference scene image; and a generation module 720, used to generate a target image containing the entity to be generated by using an image editing model based on the reference scene image and the conditional text; wherein the image editing model is trained using the training method described in any of the above embodiments.

[0163] It should be noted that the explanation of the aforementioned image editing method embodiment also applies to the image editing device of this embodiment, and therefore will not be repeated here.

[0164] In this embodiment, by using the image editing model trained by the above training method based on the reference scene image and conditional text, a target image containing the entity to be generated is generated. This ensures that the target image not only meets the conditional text description but also has scene geometry consistent with the reference scene image, thereby improving the image generation quality.

[0165] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0166] Figure 8 illustrates a schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0167] As shown in Figure 8, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. RAM 803 can also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. I / O (Input / Output) interface 805 is also connected to bus 804.

[0168] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0169] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the training method for an image editing model. For example, in some embodiments, the training method for an image editing model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the training method for the image editing model described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform a training method for an image editing model by any other suitable means (e.g., by means of firmware).

[0170] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0175] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0176] For example, the structure of the electronic device used to implement the above-described image editing method embodiment is similar to that of the above-described electronic device, and therefore will not be described again here.

[0177] According to embodiments of this application, this application also provides a computer program product, which, when executed by an instruction processor, performs the training method for the image editing model or the image editing method proposed in the above embodiments of this application.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A training method for an image editing model, comprising: An initial image editing model is adopted, and an intermediate feature vector is generated based on a sample image, a target scene image, and conditional text. The target scene image and the sample image containing the target entity are images from different perspectives within the same scene, and the conditional text describes the geometric relationship of the target entity in the target scene image. A joint attention map of the sample image and the target scene image is determined based on the intermediate feature vector. A first cross-view attention map of the sample image and a second cross-view attention map of the target scene image are determined based on the joint attention map. The initial image editing model is trained based on the differences between the first cross-view attention map and the first mask image corresponding to the sample image, and the differences between the second cross-view attention map and the second mask image corresponding to the target scene image, to obtain a trained image editing model. The first mask image and the second mask image are used to quantify the geometric consistency between the sample image and the target scene image.

2. The method as described in claim 1, wherein, The step of determining the joint attention map of the sample image and the target scene image based on the intermediate feature vector includes: extracting a first visual feature vector of the sample image and a second visual feature vector of the target scene image from the intermediate feature vector; and determining the joint attention map based on the first visual feature vector and the second visual feature vector.

3. The method as described in claim 2, wherein, Determining the joint attention map based on the first visual feature vector and the second visual feature vector includes: concatenating the first visual feature vector and the second visual feature vector to obtain a concatenated feature vector; and performing self-attention enhancement on the concatenated feature vector to obtain the joint attention map.

4. The method of claim 3, wherein, The first visual feature vector includes a first visual sub-vector, and the second visual feature vector includes a second visual sub-vector. Determining the first cross-view attention map of the sample image based on the joint attention map includes: determining the first attention intensity of the first visual sub-vector to the second visual sub-vector and the second attention intensity of the second visual sub-vector to the first visual sub-vector based on the indices of the first visual sub-vector and the second visual sub-vector in the concatenated feature vector; determining the cross-view attention intensity of the first visual sub-vector and the target scene image based on the first attention intensity and the second attention intensity; and sorting the cross-view attention intensities based on the spatial position information of the first visual sub-vector to obtain the first cross-view attention map.

5. The method of claim 4, wherein, Determining the cross-view attention intensity between the first visual sub-vector and the target scene image based on the first attention intensity and the second attention intensity includes: obtaining a first average intensity based on the average of the first attention intensity of the first visual sub-vector to all second visual sub-vectors; obtaining a second average intensity based on the average of the second attention intensity of all second visual sub-vectors to the first visual sub-vector; and determining the cross-view attention intensity based on the first average intensity and the second average intensity.

6. The method of claim 1, wherein, The first position point in the sample image is matched with the second position point in the target scene image. The first mask image and the second mask image are obtained in the following manner: an initialized third mask image and a fourth mask image are obtained; wherein the size of the third mask image is the same as the size of the sample image, and the size of the fourth mask image is the same as the size of the target scene image; the center point of the target radial kernel is translated and aligned to the first position point in the third mask image, and the elements of the target radial kernel are added to the corresponding elements of the third mask image to obtain the first mask image; the center point of the target radial kernel is translated and aligned to the second position point in the fourth mask image, and the elements of the target radial kernel are added to the corresponding elements of the fourth mask image to obtain the second mask image.

7. The method of claim 1, wherein, The target scene image is obtained by: removing the target entity from the sample image to obtain an initial scene image; generating a multi-view image sequence of the same scene based on the initial scene image; wherein the multi-view image sequence includes candidate scene images with different viewpoints from the sample image; performing sparse keypoint matching between the candidate scene images and the sample image to obtain matching information; and determining the target scene image from the candidate scene images based on the matching information.

8. The method of claim 7, wherein, The step of determining the target scene image from the candidate scene images based on the matching information includes: determining the number of position points that match between the candidate scene image and the sample image based on the matching information; and determining the target scene image from the candidate scene images based on the number of matching position points.

9. The method of claim 1, wherein, The step of training the initial image editing model based on the difference between the first cross-view attention map and the first mask image corresponding to the sample image, and the difference between the second cross-view attention map and the second mask image corresponding to the target scene image, to obtain a trained image editing model, includes: determining a first attention loss based on the difference between the first cross-view attention map and the first mask image; determining a second attention loss based on the difference between the second cross-view attention map and the second mask image; determining a geometrically guided attention loss based on the first attention loss and the second attention loss; and training the initial image editing model based on the geometrically guided attention loss to obtain a trained image editing model.

10. The method of claim 9, wherein, The step of training the initial image editing model based on the geometrically guided attention loss to obtain a trained image editing model includes: determining a noise prediction loss based on the difference between the predicted noise and the actual noise of the initial image editing model; and training the initial image editing model based on the geometrically guided attention loss and the noise prediction loss to obtain a trained image editing model.

11. An image editing method, comprising: A reference scene image and conditional text are obtained; wherein the conditional text is used to describe the geometric relationship of the entity to be generated in the reference scene image; based on the reference scene image and the conditional text, an image editing model is used to generate a target image containing the entity to be generated; wherein the image editing model is trained using the method described in any one of claims 1-10.

12. A training device for an image editing model, comprising: A first generation module is used to generate an intermediate feature vector based on a sample image, a target scene image, and conditional text using an initial image editing model; wherein the target scene image and the sample image containing the target entity are images from different perspectives within the same scene, and the conditional text is used to describe the geometric relationship of the target entity in the target scene image; a first determination module is used to determine a joint attention map of the sample image and the target scene image based on the intermediate feature vector; a second determination module is used to determine a first cross-view attention map of the sample image and a second cross-view attention map of the target scene image based on the joint attention map; a training module is used to train the initial image editing model based on the difference between the first cross-view attention map and a first mask image corresponding to the sample image, and the difference between the second cross-view attention map and a second mask image corresponding to the target scene image, to obtain a trained image editing model; wherein the first mask image and the second mask image are used to quantify the geometric consistency between the sample image and the target scene image.

13. An image editing apparatus, comprising: An acquisition module is used to acquire a reference scene image and conditional text; wherein the conditional text is used to describe the geometric relationship of the entity to be generated in the reference scene image; a generation module is used to generate a target image containing the entity to be generated by using an image editing model based on the reference scene image and the conditional text; wherein the image editing model is trained using the method described in any one of claims 1-10.

14. An electronic device comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-11.

Images