Image migration method, device, equipment, storage medium and program product

Abstract

The present application discloses an image migration method, an image migration device, an image migration equipment, a storage medium and a computer program product, and relates to the technical field of computer vision. In the present application, a fine-grained image migration method based on a diffusion model is proposed. By performing semantic matching and appearance migration in the pixel space of the diffusion model, the detail consistency between the original image and the target image is effectively maintained, and detail loss is avoided, thereby improving the image migration accuracy; through the deviation correction method of the latent space based on feature offset processing, it is ensured that the structure and semantic features of the image are effectively retained during the migration process, achieving a high-fidelity image migration effect, and enhancing structural integrity and information fidelity.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision, and in particular to an image migration method, an image migration apparatus, an image migration device, a storage medium and a computer program product. Background Art

[0002] I2I (Image-to-Image Translation) is a core task in computer vision, and its main purpose is to convert images in the source domain into corresponding images in the target domain.

[0003] In I2I research, an important subtask is appearance transfer, whose main goal is to transfer the visual aesthetic attributes of the target image to the original image while retaining the structural characteristics of the original image.

[0004] Traditional I2I models based on generative adversarial networks face challenges in adapting to different image domains, while the emergence of diffusion models provides higher flexibility and a wider range of applications.

[0005] However, there are still problems with accurate conversion during image-to-image translation, especially the inability to preserve and maintain consistency of fine-grained details, resulting in detail loss during image migration.

[0006] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0007] The main purpose of the present application is to provide an image migration method, an image migration device, an image migration equipment, a storage medium and a computer program product, aiming to solve the technical problem that image migration is not accurate enough.

[0008] To achieve the above object, the present application proposes an image migration method, which includes:

[0009] Acquire a given original image and a target image, as well as a first characteristic image of the original image in a pixel space and a second characteristic image of the target image in a pixel space;

[0010] Based on the semantic matching of the first DIFT feature and the second DIFT feature, the appearance of the target image is transferred to the original image to obtain a third feature image in the pixel space; wherein the first feature image and the second feature image are input into the DIFT diffusion model to obtain the first DIFT feature and the second DIFT feature respectively; wherein the DIFT feature is a Diffusion Features diffusion feature.

[0011] The third feature image is subjected to feature shift processing and denoising propagation processing in a latent space to obtain an edited image.

[0012] In one embodiment, the step of migrating the appearance of the target image to the original image based on the semantic matching of the first DIFT feature and the second DIFT feature to obtain a third feature image in the pixel space includes:

[0013] A mapping matching relationship is obtained by calculating a point-to-point cosine distance between the first DIFT feature and the second DIFT feature, wherein the mapping matching relationship is used to match the target feature in the second feature image in the first feature image;

[0014] Determine a cross-attention mask of the target feature in the first feature image, wherein the cross-attention mask is obtained by a cross-attention layer in a diffusion model and is obtained by thresholding;

[0015] Based on the cross-attention mask, the appearance of the target image is transferred to the original image to obtain a third feature image in pixel space.

[0016] In one embodiment, referring to formula (1), the third characteristic image in the pixel space is calculated:

[0017] Formula (1)

[0018] in, is the first feature image, is the second feature image, is the third feature image, The first feature image Cross-attention mask of target features in , represents the migration weight, Indicates the mapping matching relationship. In the first feature image The second feature image is obtained by matching The target features in .

[0019] In one embodiment, the step of performing feature shift processing on the third feature image includes:

[0020] Determine a target feature image of the original image in a latent space, and a first noise pointing to a feature state of the target feature image;

[0021] Based on the appearance difference between the third feature image and the first feature image of the original image in the pixel space, converting the third feature image into a fourth feature image, and determining a second noise pointing to a feature state of the fourth feature image;

[0022] Linearly combining the target feature image and the fourth feature image to obtain a shifted feature image in a latent space;

[0023] The first noise and the second noise are linearly combined to obtain offset feature noise corresponding to the offset feature image in the latent space.

[0024] In one embodiment, referring to formula (2), the third feature image is converted into a fourth feature image:

[0025] Formula (2)

[0026] in, is the fourth feature image, is the preset coefficient, is the first feature image, is the third feature image, represents the migration difference between appearances, is the target feature image of the original image in the latent space, To point to the target feature image The first noise of the characteristic state.

[0027] In one embodiment, the step of performing denoising and propagation processing on the third feature image in the latent space includes:

[0028] The offset feature image in the latent space and the offset feature noise corresponding to the offset feature image are input into the denoising propagation process of the diffusion model, and the offset feature in the diffusion process is recursively obtained until the offset feature is close to the target feature migrated in the target image.

[0029] In addition, to achieve the above-mentioned purpose, the present application also proposes an image migration device, the image migration device comprising

[0030] An acquisition module, used to acquire a given original image and a target image, as well as a first characteristic image of the original image in a pixel space and a second characteristic image of the target image in a pixel space;

[0031] A matching module, configured to transfer the appearance of the target image to the original image based on semantic matching of the first DIFT feature and the second DIFT feature, so as to obtain a third feature image in a pixel space; wherein the first feature image and the second feature image are input into a DIFT diffusion model to obtain a first DIFT feature and a second DIFT feature respectively;

[0032] A generating module is used to perform feature shift processing and denoising propagation processing in a latent space on the third feature image to obtain an edited image.

[0033] In addition, to achieve the above objectives, the present application also proposes an image migration device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image migration method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and stores a computer program on the storage medium. When the computer program is executed by a processor, the steps of the image migration method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the image migration method described above are implemented.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] In this application, a fine-grained image migration method based on a diffusion model is proposed. By performing semantic matching and appearance migration in the pixel space of the diffusion model, the detail consistency between the original image and the target image is effectively maintained, detail loss is avoided, and the image migration accuracy is improved; through the deviation correction method of the latent space based on feature offset processing, the structure and semantic features of the image are effectively preserved during the migration process, achieving a high-fidelity image migration effect and enhancing structural integrity and information fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the image migration method of the present application;

[0041] Figure 2 This is a schematic diagram of the application of the image migration method of this application;

[0042] Figure 3 A schematic diagram of a potential bias correction process for the image migration method of the present application;

[0043] Figure 4 This is a schematic diagram of the module structure of the image migration device according to an embodiment of the present application;

[0044] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the image migration method in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Before describing the detailed implementation of the present application, the diffusion model is explained as follows:

[0049] Diffusion model is a kind of generative model that can generate new data samples by simulating the step-by-step noise addition and denoising process of data (such as images). The process of reconstructing an image mainly consists of two stages: forward diffusion process (noise addition) and backward diffusion process (noise removal).

[0050] During the forward diffusion process, a clear image is gradually transformed into an image that is almost completely noisy by adding Gaussian noise. This process can be viewed as a known Markov chain, where the noise addition at each step follows a certain probability distribution, usually a fixed variance multiplied by the current state of the image and added to the image. As the process proceeds, the noise level of the image gradually increases and eventually approaches a standard Gaussian distribution.

[0051] During the back diffusion process, the model tries to reverse the forward diffusion process, that is, starting from an image composed entirely of noise, and gradually removing the noise to restore a clear image. This process is not known, so a trained neural network is needed to learn how to predict and remove noise from noisy images. This neural network usually uses the architecture of U-Net, which is able to capture long-range dependencies of images. During the training process, the model learns to predict the noise added to the image at each time step. This is achieved by minimizing the difference between the noise predicted by the neural network and the actual noise added. Once the training is complete, the model can use this learned denoising process to reconstruct the image from scratch, that is, starting from a random noise image, gradually generating a clear image through multiple denoising steps.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an image migration device, etc. The following takes the image migration device as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides an image migration method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the image migration method of the present application.

[0054] In this embodiment, the image migration method includes steps S10 to S30:

[0055] Step S10, obtaining a given original image and target image, as well as a first characteristic image of the original image in a pixel space and a second characteristic image of the target image in a pixel space;

[0056] It should be noted that the original image is the image waiting to be changed in appearance, such as Figure 2 The input image in ; the target image is the image to which the appearance will be transferred, such as Figure 2 The target image in .

[0057] In one embodiment, after obtaining a given original image and a target image, the denoising capability of the diffusion model can be used to extract image features to obtain a feature image in the latent space, such as Figure 2 As shown in , the target feature image of the original image in the latent space is obtained through the Null-text inverse process (inverse transformation of the real image back to the representation in the latent space) and the DDIM inverse process (prediction process). (where the feature image of the target image in the latent space is Figure 2 ), a first feature image of the original image in the pixel space and a second feature image of the target image in the pixel space are predicted from the feature image of the latent space, where, for example, the first feature image is Figure 2 In , the second feature image is Figure 2 In .

[0058] Step S20, based on the semantic matching of the first DIFT feature and the second DIFT feature, the appearance of the target image is transferred to the original image to obtain a third feature image in the pixel space; wherein the first feature image and the second feature image are input into the DIFT diffusion model to obtain the first DIFT feature and the second DIFT feature respectively;

[0059] It should be noted that DIFT features are Diffusion Features diffusion features, and the first feature image and the second feature image can be input into the DIFT diffusion model to extract the corresponding first DIFT features and the second DIFT features respectively.

[0060] The pixel space is The space can be used to represent the structural information and fine-grained features of the image, such as Figure 2 As shown, this can be done by The appearance transfer from the source image to the target image is achieved based on semantic matching in the space.

[0061] In a feasible implementation manner, the step S20 includes:

[0062] A mapping matching relationship is obtained by calculating a point-to-point cosine distance between the first DIFT feature and the second DIFT feature, wherein the mapping matching relationship is used to match the target feature in the second feature image in the first feature image;

[0063] Determine a cross-attention mask of the target feature in the first feature image, wherein the cross-attention mask is obtained by a cross-attention layer in a diffusion model and is obtained by thresholding;

[0064] Based on the cross-attention mask, the appearance of the target image is transferred to the original image to obtain the third feature image in the pixel space.

[0065] In this embodiment, first, the semantic alignment between the original image and the target image is found, that is, the visual content in the image is matched with its corresponding semantic description so that the machine can understand the content and meaning of the image. Specifically, the point-to-point cosine distance between the first DIFT feature and the second DIFT feature is calculated and sorted to determine the point-to-point matching relationship, which can be recorded as , thereby forming a semantic relevance map.

[0066] Then, a target mask-based transfer is performed to transform the appearance of the main object while preserving the background information. Figure 2 The cross attention shown in the figure is thresholded to obtain the target mask during the generation process of the diffusion model. In addition, the main object can be the object that mainly generates the occlusion, that is, the object that needs to be changed in the target image. Specifically, the cross attention mask of the target feature is used to transfer the feature, so as to achieve the fine-grained appearance migration from the target image to the original image.

[0067] In a feasible implementation, referring to formula (1), the third characteristic image in the pixel space is calculated:

[0068] Formula (1)

[0069] in, is the first feature image, is the second feature image, is the third feature image, The first feature image Cross-attention mask of target features in , represents the migration weight, Indicates the mapping matching relationship. In the first feature image The second feature image is obtained by matching The target features in .

[0070] The above migration method not only ensures accurate semantic matching, but also ensures the natural transition and background integrity during the transfer process.

[0071] Furthermore, the above migration process is further simplified as follows:

[0072] Formula (3)

[0073] in Indicates from arrive transformation.

[0074] Step S30, performing feature shift processing and denoising propagation processing in the latent space on the third feature image to obtain an edited image.

[0075] exist After the appearance migration is completed in the space, the migrated features need to be converted back to the original diffusion space, that is, from the pixel space to the latent space. This conversion process is called feature migration. The purpose of this process is to make the migrated features more smoothly integrated into the original diffusion process, which is convenient for the subsequent image iterative generation process.

[0076] It should be noted that Space can refer to image space, pixel space, Space can refer to latent space or noisy space.

[0077] In a feasible implementation manner, the step S30 includes:

[0078] Determine a target feature image of the original image in the latent space, and a first noise pointing to a feature state of the target feature image;

[0079] Based on the appearance difference between the third feature image and the first feature image of the original image in the pixel space, the third feature image is converted into a fourth feature image, and a second noise pointing to a feature state of the fourth feature image is determined;

[0080] Linearly combine the target feature image and the fourth feature image to obtain a shifted feature image in the latent space;

[0081] The first noise and the second noise are linearly combined to obtain the offset feature noise corresponding to the offset feature image in the latent space.

[0082] Reference Figure 3 In one embodiment, by converting the original image into a target feature image in the latent space Input into the U-Net model to extract the target feature image The noise of the current feature state can be expressed as the first noise .

[0083] The third feature image The first feature image in pixel space with the original image The difference in appearance between , based on the appearance difference , the third feature image Convert to the fourth feature image Similarly, the fourth feature image can be Input into the U-Net model to extract the fourth feature image The noise of the current feature state can be expressed as the second noise .

[0084] The fourth feature image can be obtained by referring to the following formula: and target feature image Perform linear combination to obtain a smoother feature representation, that is, to obtain the offset feature image of the latent space :

[0085]

[0086] in, Represents the first balance coefficient.

[0087] The first noise can be referred to the following formula: And the second noise Perform linear combination to obtain the offset feature noise corresponding to the offset feature image in the latent space :

[0088] Formula (5)

[0089] in, Represents the second balance coefficient.

[0090] In a feasible implementation, referring to formula (2), the third feature image is converted into a fourth feature image:

[0091] Formula (2)

[0092] in, is the fourth feature image, is the preset coefficient, is the first feature image, is the third feature image, represents the migration difference between appearances, is the target feature image of the original image in the latent space, To point to the target feature image The first noise of the characteristic state.

[0093] Therefore, the third feature image obtained after migration can be expressed as Converted into a new feature representation, that is, its corresponding potential feature, that is, the fourth feature image .

[0094] In a feasible implementation, the step of performing denoising and propagation processing on the third feature image in the latent space includes:

[0095] The offset feature image in the latent space and the offset feature noise corresponding to the offset feature image are input into the denoising propagation process of the diffusion model, and the offset features in the diffusion process are recursively obtained until the offset features are close to the target features migrated in the target image.

[0096] In the diffusion process, in order to keep consistent with the target domain, the offset feature image is processed by feature offset (i.e., linear combination and mixture of features and noise) and the corresponding offset characteristic noise , is input into the denoising propagation process of the diffusion model to generate the initial features for the next step , thus preparing for the next denoising process. The offset feature sequence in the entire diffusion process can be expressed as , among which, in Generated on the basis of ,exist Generated on the basis of , and so on, in this process, gradually approaching the target domain, such as Figure 2The appearance features in the target image shown. This process is repeated throughout the denoising path until the final edited image is generated. Among them, the features at each step are closely aligned with the target domain, thereby optimizing the details and consistency of the image, while ensuring that the occluded objects can reflect the appearance features of the target image as much as possible, reducing noise and unnecessary disturbances.

[0097] The diffusion model uses an iterative denoising process to reconstruct the image from the initial random noise. Among them, the DDIM method speeds up the sampling process while ensuring the high quality of the generated image. In the DDIM generation process, for a given generated image such as Figure 2 The input image in , as well as the time step t and variance during the diffusion process , the noisy feature at this time can be expressed as including the current prediction and pointing The noise composition is Add noise to get . It can be described by the following formula:

[0098] Formula (6)

[0099] In order to further analyze the diffusion process, substituting equations (2), (3), and (5) into equation (6) produces a more intuitive expression as shown below:

[0100] Formula (7)

[0101] in, Representatives in The shift in space to the target domain, the noise term point to , represents the noise term used to adapt the migration process.

[0102] The above process reveals that appearance differences are smoothly transferred to the feature space and effectively processed throughout the diffusion process, which ensures that the structural integrity of occluded objects can be maintained and the changes in appearance can be precisely controlled during the conversion of images from the source domain to the target domain.

[0103] As mentioned above, in this application, a variety of editing results can be generated according to the appearance of the input target to meet the diverse needs of occlusion generation; users can select a specific target image and accurately transfer its appearance to the occluded object, which enhances the editing flexibility of the occluded object and is more in line with the user's intention. In addition, there is no need for large-scale model training or fine-tuning, only feature migration and correction during the diffusion denoising process, which greatly reduces processing time and computing costs.

[0104] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image migration method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0105] This application also provides an image migration device, please refer to Figure 4 , the image migration device comprises:

[0106] An acquisition module 10 is used to acquire a given original image and a target image, as well as a first characteristic image of the original image in a pixel space and a second characteristic image of the target image in a pixel space;

[0107] A matching module 20 is used to transfer the appearance of the target image to the original image based on the semantic matching of the first DIFT feature and the second DIFT feature to obtain a third feature image in the pixel space; wherein the first feature image and the second feature image are input into the DIFT diffusion model to obtain the first DIFT feature and the second DIFT feature respectively;

[0108] The generating module 30 is used to perform feature shift processing and denoising propagation processing in a latent space on the third feature image to obtain an edited image.

[0109] In one embodiment, the matching module 20 is further configured to:

[0110] A mapping matching relationship is obtained by calculating a point-to-point cosine distance between the first DIFT feature and the second DIFT feature, wherein the mapping matching relationship is used to match the target feature in the second feature image in the first feature image;

[0111] Determine a cross-attention mask of the target feature in the first feature image, wherein the cross-attention mask is obtained by a cross-attention layer in a diffusion model and is obtained by thresholding;

[0112] Based on the cross-attention mask, the appearance of the target image is transferred to the original image to obtain the third feature image in the pixel space.

[0113] In one embodiment, referring to formula (1), the third characteristic image in the pixel space is calculated:

[0114] Formula (1)

[0115] in, is the first feature image, is the second feature image, is the third feature image, The first feature image Cross-attention mask of target features in , represents the migration weight, Indicates the mapping matching relationship. In the first feature image The second feature image is obtained by matching The target features in .

[0116] In one embodiment, the generating module 30 is further used for:

[0117] Determine a target feature image of the original image in the latent space, and a first noise pointing to a feature state of the target feature image;

[0118] Based on the appearance difference between the third feature image and the first feature image of the original image in the pixel space, the third feature image is converted into a fourth feature image, and a second noise pointing to a feature state of the fourth feature image is determined;

[0119] Linearly combine the target feature image and the fourth feature image to obtain a shifted feature image in the latent space;

[0120] The first noise and the second noise are linearly combined to obtain the offset feature noise corresponding to the offset feature image in the latent space.

[0121] In one embodiment, referring to formula (2), the third feature image is converted into a fourth feature image:

[0122] Formula (2)

[0123] in, is the fourth feature image, is the preset coefficient, is the first feature image, is the third feature image, represents the migration difference between appearances, is the target feature image of the original image in the latent space, To point to the target feature image The first noise of the characteristic state.

[0124] In one embodiment, the generating module 30 is further used for:

[0125] The offset feature image in the latent space and the offset feature noise corresponding to the offset feature image are input into the denoising propagation process of the diffusion model, and the offset features in the diffusion process are recursively obtained until the offset features are close to the target features migrated in the target image.

[0126] The image migration device provided by the present application adopts the image migration method in the above embodiment, which can solve the technical problem that the image migration is not accurate enough. Compared with the prior art, the beneficial effects of the image migration device provided by the present application are the same as the beneficial effects of the image migration method provided by the above embodiment, and the other technical features in the image migration device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0127] The present application provides an image migration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image migration method in the above-mentioned embodiment one.

[0128] Reference below Figure 5 , which shows a schematic diagram of the structure of an image migration device suitable for implementing the embodiment of the present application. The image migration device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The image migration device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 5As shown, the image migration device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image migration device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image migration device to communicate with other devices wirelessly or by wire to exchange data. Although the image migration device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0130] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0131] The image migration device provided by the present application adopts the image migration method in the above embodiment, which can solve the technical problem that the image migration is not accurate enough. Compared with the prior art, the beneficial effects of the image migration device provided by the present application are the same as the beneficial effects of the image migration method provided by the above embodiment, and the other technical features in the image migration device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0134] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the image migration method in the above-mentioned embodiment.

[0135] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0136] The computer-readable storage medium may be included in the image migration device; or may exist independently without being assembled into the image migration device.

[0137] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image migration device, the image migration device: obtains a given original image and a target image, and a first feature image of the original image in a pixel space and a second feature image of the target image in a pixel space;

[0138] Based on the semantic matching of the first DIFT feature and the second DIFT feature, the appearance of the target image is transferred to the original image to obtain a third feature image in the pixel space; wherein the first feature image and the second feature image are input into the DIFT diffusion model to obtain the first DIFT feature and the second DIFT feature respectively;

[0139] The third feature image is subjected to feature shift processing and denoising propagation processing in the latent space to obtain an edited image.

[0140] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0143] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image migration method, and can solve the technical problem that image migration is not accurate enough. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the image migration method provided in the above-mentioned embodiment, and will not be described in detail here.

[0144] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image migration method when executed by a processor.

[0145] The computer program product provided by the present application can solve the technical problem of inaccurate image migration. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the image migration method provided by the above embodiment, which will not be described in detail here.

[0146] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image migration method, characterized in that: The image migration method comprises: Acquire a given original image and a target image, as well as a first characteristic image of the original image in a pixel space and a second characteristic image of the target image in a pixel space; Based on the semantic matching of the first DIFT feature and the second DIFT feature, the appearance of the target image is transferred to the original image to obtain a third feature image in the pixel space; wherein the first feature image and the second feature image are input into the DIFT diffusion model to obtain the first DIFT feature and the second DIFT feature respectively; Performing feature shift processing and denoising propagation processing in a latent space on the third feature image to obtain an edited image; The step of migrating the appearance of the target image to the original image based on the semantic matching of the first DIFT feature and the second DIFT feature to obtain a third feature image in the pixel space comprises: A mapping matching relationship is obtained by calculating a point-to-point cosine distance between the first DIFT feature and the second DIFT feature, wherein the mapping matching relationship is used to match the target feature in the second feature image in the first feature image; Determine a cross-attention mask of the target feature in the first feature image, wherein the cross-attention mask is obtained by a cross-attention layer in a diffusion model and is obtained by thresholding; Based on the cross attention mask, the appearance of the target image is transferred to the original image to obtain a third feature image in pixel space; The step of performing feature offset processing on the third feature image comprises: Determine a target feature image of the original image in a latent space, and a first noise pointing to a feature state of the target feature image; Based on the appearance difference between the third feature image and the first feature image of the original image in the pixel space, converting the third feature image into a fourth feature image, and determining a second noise pointing to a feature state of the fourth feature image; Linearly combining the target feature image and the fourth feature image to obtain a shifted feature image in a latent space; Linearly combining the first noise and the second noise to obtain offset feature noise corresponding to the offset feature image in the latent space; The step of performing denoising and propagation processing on the third feature image in the latent space comprises: The offset feature image in the latent space and the offset feature noise corresponding to the offset feature image are input into the denoising propagation process of the diffusion model, and the offset feature in the diffusion process is recursively obtained until the offset feature is close to the target feature migrated in the target image.

2. The image migration method according to claim 1, characterized in that: Referring to formula (1), the third feature image in the pixel space is calculated: Formula(1) in, is the first feature image, is the second feature image, is the third feature image, The first feature image Cross-attention mask of target features in , represents the migration weight, Indicates the mapping matching relationship. In the first feature image The second feature image is obtained by matching The target features in .

3. The image migration method according to claim 1, characterized in that: Referring to formula (2), the third feature image is converted into a fourth feature image: Formula (2) in, is the fourth feature image, is the preset coefficient, is the first feature image, is the third feature image, represents the migration difference between appearances, is the target feature image of the original image in the latent space, To point to the target feature image The characteristic state of the first noise.

4. An image migration device, characterized in that: The image migration device implements the method according to claim 1, and the image migration device comprises: An acquisition module, used to acquire a given original image and a target image, as well as a first characteristic image of the original image in a pixel space and a second characteristic image of the target image in a pixel space; A matching module, configured to transfer the appearance of the target image to the original image based on semantic matching of the first DIFT feature and the second DIFT feature, so as to obtain a third feature image in a pixel space; wherein the first feature image and the second feature image are input into a DIFT diffusion model to obtain a first DIFT feature and a second DIFT feature respectively; A generating module is used to perform feature shift processing and denoising propagation processing in a latent space on the third feature image to obtain an edited image.

5. An image migration device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image migration method according to any one of claims 1 to 3.

6. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image migration method according to any one of claims 1 to 3 are implemented.

7. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the image migration method according to any one of claims 1 to 3 are implemented.

Images