Image processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses an image processing method and device, equipment, a storage medium and a program product. The image processing method comprises the following steps: performing first coding processing on a first image and a second image to obtain a first image feature and a second image feature; performing interpolation processing on the first image feature and the second image feature to obtain a first interpolation feature; and decoding the first interpolation feature to obtain a transition image, and inserting the transition image between the first image and the second image. According to the process, interpolation between the images is converted into interpolation between the image features, and compared with an original image, the information amount of the image features is greatly reduced, so that the implementation difficulty of the image interpolation process is effectively reduced, the accuracy of the generated transition image is improved, interpolation operation of any two images is realized, and the image processing efficiency is improved. And the robustness of transition image generation is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image processing method, device, equipment, storage medium and program product. BACKGROUND

[0002] Interpolation, usually referred to as interpolation, is an image processing term. The essence of interpolation is to estimate the value of an unknown position by using known data. Image interpolation refers to generating a new image by the method of "interpolation" between two input images. The image interpolation problem is similar to the fitting problem, both of which are important components of function approximation or numerical approximation. However, the difference lies in that for a given function, interpolation requires that the discrete points "lie on" the function curve to meet the constraints, while fitting hopes that the discrete points "approximate" the function curve as much as possible. SUMMARY

[0003] The embodiments of the present specification provide an image processing method, device, equipment, storage medium and program product, so as to effectively reduce the difficulty of image interpolation process implementation and improve the accuracy of the generated transition image, realize the interpolation operation of any two images.

[0004] In a first aspect, the embodiments of the present specification provide an image processing method, comprising:

[0005] performing first encoding processing on the first image and the second image to obtain first image features and second image features;

[0006] performing interpolation processing on the first image features and the second image features to obtain first interpolation features;

[0007] performing decoding processing on the first interpolation features to obtain a transition image, and inserting the transition image between the first image and the second image.

[0008] As can be seen, in the embodiments of the present specification, the first image and the second image to be interpolated are subjected to first encoding processing, and the first image features and the second image features obtained by encoding are subjected to interpolation processing to obtain first interpolation features, and then the first interpolation features are subjected to decoding processing to generate a transition image. This process converts the interpolation between images into the interpolation between image features. Compared with the original image, the information amount of the image features is greatly reduced, which effectively reduces the difficulty of image interpolation process implementation and improves the accuracy of the generated transition image, realizes the interpolation operation of any two images, and further improves the robustness of the transition image generation.

[0009] In a second aspect, the embodiments of the present specification provide an image processing device, comprising:

[0010] The encoding module is configured to perform first encoding processing on the first image and the second image to obtain first image features and second image features.

[0011] The interpolation module is configured to perform interpolation processing on the first image features and the second image features to obtain first interpolation features.

[0012] The decoding module is configured to perform decoding processing on the first interpolation features to obtain a transition image and insert the transition image between the first image and the second image.

[0013] In a third aspect, an electronic device is provided, and the electronic device includes:

[0014] a processor; and a memory arranged to store computer-executable instructions configured to be executed by the processor, the computer-executable instructions including instructions for performing the steps in the image processing method provided in the first aspect.

[0015] In a fourth aspect, a storage medium is provided, and the storage medium is configured to store computer-executable instructions, the computer-executable instructions causing a computer to execute the image processing method provided in the first aspect.

[0016] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program, the computer program being executed by a processor to implement the image processing method provided in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of one or more embodiments of the present specification or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0018] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present specification;

[0019] Figure 2 is a structural diagram of an image processing model provided by an embodiment of the present specification;

[0020] Figure 3 is a diagram of image interpolation provided by an embodiment of the present specification;

[0021] Figure 4 is a diagram of image interpolation provided by an embodiment of the present specification;

[0022] Figure 5 FIG. 1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present specification;

[0023] Figure 6 FIG. 2 is a structural schematic diagram of an image processing device according to an embodiment of the present specification;

[0024] Figure 7 FIG. 3 is a structural schematic diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present specification will be clearly and completely described in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present document.

[0026] The terms "first", "second", and the like in the specification and claims of the present document are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present specification can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0027] The image processing method, device, equipment, storage medium and program product provided by the embodiments of the present specification will be described in detail below in conjunction with the drawings, specific embodiments and application scenarios thereof.

[0028] Figure 1 An embodiment of the present application provides an image processing method, which can be executed by an electronic device, which can include a server and / or a terminal device, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the aforementioned electronic device, and the method includes the following steps:

[0029] Step S102: performing first encoding processing on the first image and the second image to obtain first image features and second image features.

[0030] The first image and the second image are images used to generate the interpolation image. After receiving the generation instruction of the interpolation image, the interpolation image (i.e., the transition image) between the styles, textures, and the like of the first image and the second image can be generated according to the styles, textures, and the like of the first image and the second image indicated by the generation instruction. The difference between the styles, textures, and the like of the first image and the second image is not specifically limited in the specification, and can be determined according to actual conditions.

[0031] The first encoding processing can be an operation of mapping a two-dimensional image to a high-dimensional latent space. Through the first encoding processing, the image can be converted into an image feature. The first image feature is an image feature obtained by performing the first encoding processing on the first image, and the second image feature is an image feature obtained by performing the first encoding processing on the second image.

[0032] It should be noted that the first encoding processing is different from the processing manner of a Convolutional Neural Network (CNN), which simultaneously performs compression and mapping processing on an image, while the first encoding processing in the specification does not include compression processing, and the size of the shape of the image feature needs to be kept consistent with the original image during the first encoding processing, which will help to ensure that the transition image obtained in the subsequent step S106 is consistent in size with the first image and the second image, and improve the effect of the transition image generated in the image processing process. The dimension parameter of the latent space (i.e., the dimension of the first image feature and the second image feature) is not specifically limited in the specification, and can be determined according to actual conditions. For example, the dimension parameter can be 1000 or 2000, etc.

[0033] In an example, the first encoding processing on the first image and the second image can be performed by an encoder in a Variational Auto-Encoders (VAE) to map the first image and the second image to the latent space to obtain the first image feature and the second image feature.

[0034] Step S104: performing interpolation processing on the first image feature and the second image feature to obtain a first interpolation feature.

[0035] The interpolation processing is used to obtain an image feature (i.e., the first interpolation feature) between the first image feature and the second image feature, and the first interpolation feature is used to represent information contained in the transition image between the first image and the second image.

[0036] The first encoding process in step S102 converts the two-dimensional first image into a high-dimensional first image feature and the two-dimensional second image into a high-dimensional second image feature. In this way, the first image feature and the second image feature remove global and detailed information from the original image (first image and second image) and retain only some important features in the original image. This makes the feature distribution represented by the first image feature and the second image feature more similar, which is beneficial to the generation of transition images between any two images.

[0037] In one example, interpolation between the first and second image features can be performed directly in the latent space. Specifically, this process can involve converting the first and second image features (both in matrix form) into corresponding vectors (i.e., row vectors), and then performing interpolation between the two transformed vectors. This specification does not specify a particular method for interpolating the first and second image features; it can be determined based on the actual situation. For example, the interpolation could be a linear interpolation operation, a nearest neighbor interpolation operation, etc.

[0038] The image processing methods described in this specification can be used in image processing models. Figure 2 A schematic diagram of the structure of an image processing model is shown, such as... Figure 2 As shown, an interpolation module (such as the first interpolation module IPM1) can be set in the image processing model to generate the first interpolation feature between two image features, so as to realize the interpolation processing of the first image feature and the second image feature.

[0039] Specifically, such as Figure 2 As shown, a linear interpolation module IPM1 can be set as the first interpolation module in the image processing model. The first image features and the second image features are input into the linear interpolation module IPM1. Then, the linear interpolation module IPM1 can perform linear interpolation calculation on the first image features and the second image features. Then, the interpolation result is processed by a multi-layer perceptron (MLP) in the linear interpolation module IPM1 to obtain the first interpolated feature F. N First interpolation feature F N The calculation method is shown in Formula 1.

[0040] F N =MLP(LinearInterpolation(N image1 N image2 )) (Formula 1)

[0041] wherein, N image1 denotes the first image feature, N image2 denotes the second image feature, F N denotes the first interpolation feature, LinearInterpolation(·) denotes a linear interpolation operation, and MLP(·) denotes processing using a multi-layer perceptron.

[0042] In order to improve the training efficiency of the image processing model, Figure 2 The first interpolation module in the image processing method can be a pre-trained model whose parameters do not need to be changed in the training process of the image processing model.

[0043] Step S106: decoding the first interpolation feature to obtain a transition image and inserting the transition image between the first image and the second image.

[0044] The decoding process can be an inverse operation of the first encoding process in step S102. Specifically, the decoding process can be inverse mapping of the second image feature in the latent space to the transition image.

[0045] In the embodiments of the present specification, the first image and the second image to be interpolated are subjected to first encoding processing, and the first image feature and the second image feature obtained by encoding are subjected to interpolation processing to obtain a first interpolation feature, and then the first interpolation feature is subjected to decoding processing to generate a transition image. This process converts the interpolation between images into the interpolation between image features. Compared with the original image, the information amount of the image feature is greatly reduced, which effectively reduces the difficulty of implementing the image interpolation process and improves the accuracy of the generated transition image, realizes the interpolation operation of any two images, and further improves the robustness of the transition image generation.

[0046] Figure 3 and Figure 4 is a schematic diagram of image interpolation using the image processing method in the present specification. As shown in Figure 3 and Figure 4 shown, Figure 3 and Figure 4 The image interpolation in the present specification is to generate a new interpolation image between two real images. The interpolation image has the texture, style and other characteristics of the two real images, and the characteristics of the interpolation image are between the two real images. The difference is that, Figure 3 is to interpolate between similar images, Figure 4 is to interpolate between images with large differences (for example Figure 4 The difference in the face angle, accessories, hairstyle in the present specification is large.

[0047] Figure 5 An application scenario diagram of the image processing method is provided, as shown inFigure 5 As shown, a user sends a service request for generating a transition image between the first image and the second image to a server through a terminal such as a mobile phone, and the server generates the transition image using the image processing method in the present specification after receiving the service request, and sends the transition image to the terminal for the user to refer.

[0048] As mentioned above, the first encoding processing can cause the image to lose some detailed information, and thus facilitate the smooth interpolation processing. However, the first image feature and the second image feature obtained by the first encoding processing can still include some detailed information. In an implementation, the first image feature and the second image feature are interpolated to obtain a first interpolation feature, including:

[0049] The first image feature and the second image feature are added with noise to obtain a first image feature added with noise and a second image feature added with noise.

[0050] The first image feature added with noise and the second image feature added with noise are interpolated to obtain the first interpolation feature.

[0051] Specifically, the first image feature and the second image feature can be added with noise first, and then the first image feature added with noise and the second image feature added with noise are interpolated to obtain the first interpolation feature. Of course, the first image and the second image can also be added with noise first, and then the first image feature and the second image feature obtained after the noise is added are interpolated to obtain the first interpolation feature.

[0052] In an example, the first image feature added with noise and the second image feature added with noise can be obtained by adding noise to the first image feature and the second image feature continuously for multiple times or directly adding noise once. The method of adding noise and the details of adding noise, such as the number of times of adding noise, the type of noise, etc., are not limited in the present specification, and can be selected according to actual conditions.

[0053] As mentioned above, the first image feature and the second image feature can be added with noise first, and then the first image feature added with noise and the second image feature added with noise are interpolated to obtain the first interpolation feature. Of course, the first image and the second image can also be added with noise first, and then the first image feature and the second image feature obtained after the noise is added are interpolated to obtain the first interpolation feature. Figure 2As shown, Gaussian noise can be added to the first image and the second image using a Gaussian noise model as a noise model. Specifically, the Gaussian noise model can first perform a first encoding operation on the first image and the second image, map the first image and the second image to a latent space, and then add Gaussian noise. The Gaussian noise model used by the first image and the second image can be the same. The manner in which the Gaussian noise model adds noise can be the same as the manner in which noise is added in the diffusion model DDPM (Denoising Diffusion Probabilistic Models). That is, the image can be mapped to a latent space by an encoder in a variational auto-encoder (VAE), and then noise can be added to the first image and the second image in the latent space. The calculation process of the first image feature can be as shown in Equation 2:

[0054] N image1 = DDPM (I1, T) (Equation 2)

[0055] where I1 represents the first image input to the Gaussian noise model, DDPM(·) represents adding Gaussian noise using the diffusion model DDPM, T represents the number of noise added, and N image1 represents the first image feature after adding noise. The shape of the first image feature can be the same size as the original image. The calculation process of the second image feature can be the same as that of the first image feature, which is not described in detail here.

[0056] In the above embodiment, although the first image feature and the second image feature obtained after adding noise lose global, detailed, and other information in the first image and the second image, the distance between the first image feature and the second image feature after adding noise in the latent space is closer, and the generation of the transition image between any first image and second image can be better realized.

[0057] In an implementation manner, the first interpolation feature is decoded to obtain the transition image, including:

[0058] Obtaining a decoding condition of the first interpolation feature;

[0059] Based on the decoding condition, the first interpolation feature is decoded using a diffusion sub-model to obtain the transition image.

[0060] The decoding condition corresponds to a condition for performing inverse operations on the aforementioned first encoding process and the noise addition process. Since the first interpolation feature is generated based on the first image and the second image after adding noise, the transition image can be generated by performing a decoding process including a noise removal process on the first interpolation feature. Specifically, the first interpolation feature can be denoised in a manner of predicting the noise added at the previous time step, or in a manner of predicting the feature at the previous time step, which is not specifically limited in the present specification.

[0061] Since the first image feature and the second image feature after adding noise lose a lot of information in the original image (the first image and the second image), the first interpolation feature also loses a lot of information in the first image and the second image. To improve the generation effect (such as realism) of the transition image, a decoding condition can be set for the decoding process, which can guide the sampling and generation of the transition image, so that the objects in the transition image generated after the decoding process can match the objects in the original image as much as possible.

[0062] In an example, part of the information (such as the position of the object in the image, texture information, etc.) can be extracted from the first image and the second image as the decoding condition to guide the decoding process. Specifically, other decoding conditions such as image category labels, keywords, etc. can also be set according to the display requirements of the transition image. The generation process of the decoding condition can refer to subsequent embodiments, which will not be described here.

[0063] Specifically, the decoding process including the denoising or noise reduction process can be implemented through an algorithm architecture as follows, which can be divided into three parts: a condition module for adding a denoising condition (i.e., a decoding condition), a diffusion module for denoising the image feature with noise in the latent space, and a decoding module for mapping the denoised image feature from the latent space back to the pixel space. The condition in the condition module can be semantic condition Semantic Map (indicating a task of generating an image through semantics), text condition Text (indicating a task of generating an image through text), description condition Representations (indicating a task of generating an image through language description), image condition Images (indicating a task of generating an image according to an image), and the like. The decoding module can be a decoder in a variational autoencoder. The structure of the diffusion module can refer to the Stable Diffusion (SD) model, which will not be described here.

[0064] For example, the decoding condition can be a semantic condition, a text condition, a description condition, an image condition, or the like. Figure 2As shown, a diffusion sub-model (Latent Diffusion Model, LDM) using the principle of the diffusion model DDPM can be set in the image processing model, and the diffusion sub-model is used to convert the first interpolation feature into a transition image and output.

[0065] The above process converts the image interpolation task into an image generation task through decoding processing of the noisy first interpolation feature using the diffusion sub-model, realizes interpolation operation of any two images based on the first image feature and the second image feature close in distance after being added with noise, and improves the robustness of the image processing model.

[0066] Since the first image feature and the second image feature lose global, detailed, and other information of the first image and the second image, in order to improve the generation effect of the transition image, part of the information of the first image and the second image can be added to the first image feature and the second image feature before decoding processing. In an implementation manner, the first interpolation feature is decoded using the diffusion sub-model based on a decoding condition to obtain the transition image, including:

[0067] The first image and the second image are subjected to second encoding processing to obtain third image features and fourth image features.

[0068] The third image features and the fourth image features are subjected to interpolation processing to obtain second interpolation features.

[0069] The first interpolation feature is adjusted based on the second interpolation feature to obtain an adjusted first interpolation feature.

[0070] The adjusted first interpolation feature is decoded using the diffusion sub-model based on a decoding condition to obtain the transition image.

[0071] The second encoding processing also includes the operation of mapping the two-dimensional image to the high-dimensional latent space, like the first encoding processing. In addition, the second encoding processing also includes a feature extraction operation on the image features obtained by mapping, that is, the third image features and the fourth image features are image features obtained by further extracting features from the image features obtained by mapping. Specifically, the third image features represent image features obtained by second encoding processing on the first image, and the fourth image features represent image features obtained by second encoding processing on the second image. The latent spaces corresponding to the first encoding processing and the second encoding processing can be different, so that the generalization ability and the robustness of the image processing method can be improved by supplementing the features of different latent spaces.

[0072] Further, after obtaining the third image feature and the fourth image feature, a method similar to the generation method of the first interpolation feature can be adopted to perform interpolation processing on the third image feature and the fourth image feature to obtain a second interpolation feature, and then the second interpolation feature can be supplemented to the first interpolation feature to adjust the first interpolation feature, and finally the transition image can be generated through the adjusted first interpolation feature.

[0073] Specifically, a weighted average can be used on the first interpolation feature and the second interpolation feature to obtain the second interpolation feature. In an example, before the first interpolation feature is decoded by using the diffusion sub-model, the first interpolation feature is adjusted based on the second interpolation feature to obtain an adjusted first interpolation feature, including:

[0074] obtaining a first weight of the first interpolation feature and a second weight of the second interpolation feature;

[0075] obtaining the adjusted first interpolation feature based on the first interpolation feature and the first weight, and the second interpolation feature and the second weight.

[0076] The first weight corresponding to the first interpolation feature and the second weight corresponding to the second interpolation feature are not specifically limited in the specification, and can be determined according to actual conditions.

[0077] To fully extract different types of information in the first image and the second image, in an implementation manner, the second encoding processing includes M encoding types, and the first image and the second image are processed by the second encoding processing to obtain third image features and fourth image features, including:

[0078] processing the first image and the second image by the second encoding processing based on the M encoding types to obtain M third image features and M fourth image features;

[0079] performing interpolation processing on the third image features and the fourth image features to obtain second interpolation features, including:

[0080] performing interpolation processing on the third image features and the fourth image features belonging to the same encoding type to obtain M second interpolation features.

[0081] Specifically, M different second encoding processing processes can be set for the first image and the second image, and each second encoding processing process can include mapping the first image and the second image to different latent spaces, and performing feature extraction operations on the mapped image features to respectively implement different types of second encoding processing on the first image and the second image in different latent spaces. Each second encoding processing process can correspond to a latent space, and each second encoding processing process can correspond to a set of third image features and fourth image features.

[0082] Further, the third image feature and the fourth image feature corresponding to each second encoding processing process can be respectively interpolated to obtain a second interpolation feature corresponding to the second encoding processing process. In an example, the second encoding processing process can include two types: mapping the first image and the second image to a feature space corresponding to an instance feature (a feature of an object such as a face in an image), and extracting the instance feature from the mapped image feature; and mapping the first image and the second image to a feature space corresponding to a global feature, and extracting the global feature from the mapped image feature.

[0083] Vision Transformer (ViT) is a neural network model based on self-attention mechanism, which uses attention to capture global context information to establish long-distance dependence on the target, thereby improving the representation ability of features. As shown in Figure 2 ViT module can be set in the image processing model, and the ViT module can project the first image and the second image into a global space through second encoding processing, and then extract global features of the first image and the second image in the global space to obtain a third image feature representing global information of the first image and a fourth image feature representing global information of the second image. The calculation process of the third image feature corresponding to the global space can be as shown in formula 3:

[0084] V image1 = Vit(I1) (Formula 3)

[0085] wherein, I1 represents the first image input into the ViT module, V image1 represents the third image feature corresponding to the global space, and Vit(·) represents extracting the global feature. The calculation process of the fourth image feature corresponding to the global space can be the same as that of the third image feature corresponding to the global space, which is not described in detail here. Like the first image feature and the second image feature, the third image feature and the fourth image feature corresponding to the global space can have the same size as the shape of the original image.

[0086] Segment Anything model (SAM) is a model with wide functions in the field of image segmentation, which can quickly adapt to many existing and new segmentation tasks (such as edge detection, target proposal generation, instance segmentation, and object segmentation from free text). As shown in Figure 2As shown, the SAM module can be configured in the image processing model, and the SAM module can project the first image and the second image to the instance space through the second encoding processing, and then can extract the instance features of the first image and the second image in the instance space to obtain the third image features representing the instance information of the first image and the fourth image features representing the instance information of the second image. Specifically, the SAM module can segment the object from the image features in the instance space, and then obtain the instance features as the third image features or the fourth image features. The calculation process of the third image features corresponding to the instance space can be as shown in formula 4:

[0087] S image1 =SAM(I1) (Formula 4)

[0088] wherein, I1 represents the first image input to the SAM module, S image1 represents the third image features corresponding to the instance space, SAM(·) represents extracting instance features in the instance space. The calculation process of the fourth image features corresponding to the instance space can be the same as that of the third image features corresponding to the instance space, which is not described in detail here. Like the first image features and the second image features, the third image features and the fourth image features corresponding to the instance space can be the same size as the shape of the original image.

[0089] After obtaining the third image features and the fourth image features corresponding to the global space and the instance space, similar to the calculation method of the first interpolation features, the third image features and the fourth image features of the global space and the instance space can be calculated by linear interpolation based on formula 5 and formula 6, and then the corresponding interpolation features can be obtained using the multi-layer perception (MLP).

[0090] F V =MLP(LinearInterpolation(V image1 ,V image2 )) (Formula 5)

[0091] wherein, V image1 represents the third image features corresponding to the global space, V image2 represents the fourth image features corresponding to the global space, F V represents the second interpolation features corresponding to the global space, LinearInterpolation(·) represents linear interpolation operation, and MLP(·) represents processing using multi-layer perception.

[0092] F S =MLP(LinearInterpolation(S image1 ,S image2 )) (Formula 6)

[0093] wherein S image1 denotes a third image feature corresponding to the instance space, S image2 denotes a fourth image feature corresponding to the instance space, F S denotes a second interpolation feature corresponding to the instance space, LinearInterpolation(·) denotes a linear interpolation operation, and MLP(·) denotes processing using a multi-layer perceptron.

[0094] In the case where the feature space comprises a global space and an instance space, the calculation manner of the adjusted first interpolation feature can be as shown in Formula 7.

[0095] F all = 1 / 3(a*F N +b*F V +c*F S ) (Formula 7)

[0096] wherein a, b, c are hyperparameters, F N denotes the first interpolation feature, F V denotes a second interpolation feature corresponding to the global space, F S denotes a second interpolation feature corresponding to the instance space, F all denotes the adjusted first interpolation feature. The first interpolation feature can have the same size as the shape of the original image.

[0097] In the embodiments of the present specification, the third image feature is extracted from the first image, the fourth image feature is extracted from the second image, the second interpolation feature is generated through interpolation processing of the third image feature and the fourth image feature, and then the first interpolation feature is adjusted using the second interpolation feature to obtain the adjusted first interpolation feature. This process can increase the detail information of the transition image by supplementing the first interpolation feature based on the second interpolation feature representing the detail information of the first image and the second image, and thus can improve the effect of the generated transition image.

[0098] In an implementation manner, the decoding condition comprises text information obtained from the first image and the second image, and the process of obtaining the text information comprises the following steps:

[0099] textually describing the first image and the second image to obtain first image text and second image text;

[0100] inputting the first image text and the second image text into a language model to obtain the text information.

[0101] The image description is output according to the provided image. The image description task can be divided into two parts: one part is to encode the image to be described, and the other part is to generate text according to the information obtained after encoding. The first image text is the text obtained by text description of the first image, and the second image text is the text obtained by text description of the second image. The manner of implementing image description is not limited in the specification, and can be determined according to actual conditions.

[0102] As shown in Figure 2 An image description module (image caption) can be set in the image processing model, which is used to convert the image visual features extracted by the computer into high-level semantic information. The image description module can be a pre-trained model whose parameters do not need to be changed in the training process of the image processing model. Specifically, the same image description module can be used for image description of the first image and the second image to obtain the first image text corresponding to the first image and the second image text corresponding to the second image, respectively.

[0103] After obtaining the first image text and the second image text, a language model can be used to extract more accurate text information from the first image text and the second image text. In an example, a large language model can be used as the language model, which can be GPT, LLaMA or ChatGLM, etc. Since the large language model has the advantages of fast training speed, good interpretability and strong generalization ability, the large language model can be used to integrate the information in the first image text and the second image text to obtain the text information. Specifically, the first image text and the second image text can be input into the large language model, and the output of the large language model can be used as the text information.

[0104] The diffusion sub-model decodes the first interpolation feature based on the decoding condition including the text information. The process can be a vector (text information corresponding to a text vector) obtained by encoding the text information using a conditional encoder, and the vector is sent into the U-Net network of the diffusion sub-model as a decoding condition based on the cross attention mechanism (cross attention). The U-Net network is the main body of the diffusion sub-model, which is used to generate the transition image guided by the condition.

[0105] In the embodiments of the present specification, the first image and the second image are first described by text, and then the first image text and the second image text obtained by the text description process are integrated by the language model to obtain the text information. Since the language model has the advantages of strong data processing ability and high accuracy of generated results, the process based on the language model enhances the description ability of the text information to the first image and the second image, and thus the effect of the transition image generated by the text information can be improved.

[0106] In an implementation manner, the training process of the diffusion sub-model comprises the following steps:

[0107] The first sample image and the second sample image are subjected to first encoding processing to obtain first sample image features and second sample image features;

[0108] After adding first noise to the first sample image features and the second sample image features, the first sample image features and the second sample image features are subjected to interpolation processing to obtain first sample interpolation features;

[0109] Based on the obtained decoding condition, the first sample interpolation features are subjected to decoding processing using the diffusion sub-model to obtain a sample transition image, and second noise removed by the decoding processing corresponding to the sample transition image is determined;

[0110] According to the difference between the first noise and the second noise, a first loss is constructed, and the diffusion sub-model is trained according to the first loss to obtain a trained diffusion sub-model.

[0111] The first sample image and the second sample image can be any selected natural image (such as an image obtained by shooting) or a computer-generated image. The source of the first sample image and the second sample image is not limited in the present specification and can be selected according to actual conditions. Like the first image and the second image, the object in the first sample image and the second sample image is not limited in the present specification and can be selected according to actual conditions. The object can be a person, an object, an animal, etc. Due to the randomness of the original sample image (the first sample image and the second sample image) and the non-annotated characteristics of the original sample image, the original sample image selected in the present specification has a high degree of selection space.

[0112] Referring to the generation process of the transition image described above, for each set of first sample image and second sample image, a sample transition image can be generated. Specifically, the first sample image and the second sample image can be subjected to first encoding processing to obtain first sample image features corresponding to the first sample image and second sample image features corresponding to the second sample image, and then the first sample image features and the second sample image features can be subjected to interpolation processing after adding noise to obtain first sample interpolation features. The sample transition image can be obtained by using the diffusion sub-model to subject the first sample interpolation features to decoding processing.

[0113] When the first sample interpolation features are subjected to decoding processing, a decoding condition for controlling the decoding processing process of the first sample interpolation features can be obtained (referring to the method for obtaining the decoding condition of the first interpolation features described above), and based on the decoding condition, the diffusion sub-model is used to subject the first sample interpolation features to decoding processing to obtain the sample transition image.

[0114] During the training process, the parameters of the diffusion sub-model can be adjusted according to the first loss, which can be determined according to the difference between the first noise added in the first encoding process and the second noise removed in the decoding process. Specifically, the first loss of the diffusion sub-model can be as shown in formula 8:

[0115] L LDM : min((||ε-ε θ (F all ,t,C)||2) 2 ) (Formula 8)

[0116] Wherein, L LDM represents the optimization target of the diffusion sub-model, ε represents the noise added to the first sample image and the second sample image, ε θ represents the noise prediction network, t represents the number of steps of adding noise to the first sample image and the second sample image, C represents the encoding data corresponding to the decoding condition, F all represents the first interpolation feature.

[0117] In the embodiments of the present specification, the diffusion sub-model is trained based on the first loss, and then the generation of the transition image between any two images can be realized based on the trained diffusion sub-model, thereby improving the robustness of the image processing process.

[0118] In an implementation manner, the same set of first sample images and second sample images corresponds to a plurality of sample transition images, and the diffusion sub-model is trained according to the first loss to obtain the trained diffusion sub-model, including:

[0119] The decoding conditions corresponding to the same set of first sample images and second sample images and the sample transition images are respectively encoded to obtain sample text features of the decoding conditions and sample image features of the sample transition images;

[0120] Based on the matching degree of the sample text features and the sample image features, a ranking sub-model is used to determine a target transition image from the sample transition images;

[0121] According to the difference between the sample image features and the sample text features corresponding to the target transition image, a second loss is constructed;

[0122] The diffusion sub-model and the ranking sub-model are jointly trained according to the first loss and the second loss to obtain the trained diffusion sub-model and the ranking sub-model.

[0123] During the training of the diffusion sub-model, the effect of the sample transition images output by the diffusion sub-model can be evaluated based on the decoding conditions (usually text) to determine whether the sample transition images output by the diffusion sub-model meet the requirements. Specifically, the sample transition images and decoding conditions can be encoded separately, and the matching operation between the sample text features corresponding to the decoding conditions and the sample image features corresponding to the sample transition images can be used to determine whether the sample transition images and decoding conditions match.

[0124] Because the images generated by the diffusion sub-model are diverse, for the same set of first and second sample images, the diffusion sub-model can generate multiple sample transition images. Therefore, based on the degree of matching between each sample transition image and the decoding conditions, the ranking sub-model can determine the target transition image from these multiple sample transition images. Furthermore, based on the difference between the sample image features and sample text features of the target transition image, the effectiveness of the image output by the diffusion sub-model can be determined. The operation of this ranking sub-model can include sorting the degree of matching between each sample transition image and the decoding conditions to determine the target transition image, which has the best match between its sample image features and the decoding conditions among the multiple sample transition images.

[0125] like Figure 2 As shown, a ranking sub-model can be set in the image processing model. The text encoding module within this sub-model encodes the decoding conditions controlling image generation, and the image encoding module encodes multiple transition images output by the diffusion sub-model. Then, the ranking module can be used to match the obtained sample text features with multiple sample image features and rank them based on the matching results. The difference between the sample image features and sample text features of the target transition image determines whether the generated sample transition image meets the requirements. The loss constructed based on the difference between the sample image features and sample text features of the target transition image is the second loss corresponding to the ranking sub-model. The calculation process of this second loss can be shown in Equation 9.

[0126] L sorting =Contrastive(CLIP) image-encoder (IP image ), C s ) (Formula 9)

[0127] Among them, IP image CLIP represents the sample transition image output by the diffusion sub-model. image-encoder (·) indicates an encoding operation, C s Represents the features of the sample text, Contrastive(·) denotes the calculation of the contrastive loss, L sorting This represents the second loss of the ranking sub-model.

[0128] Furthermore, the diffusion sub-model and the ranking sub-model can be jointly trained, and the total loss corresponding to this joint training can be as follows:

[0129] Loss = L LDM +L sorting (Formula 10)

[0130] Where Loss represents the total target loss, L LDM L represents the first loss of the diffusion sub-model. sorting This represents the second loss of the ranking sub-model.

[0131] for Figure 2 In the image processing model described in Formula 10, the target loss can be used as the total loss of the entire image processing model, and the parameters of each module of the image processing model can be adjusted accordingly. To improve the training efficiency of the image processing model, Figure 2 The text encoding module and image encoding module in the model can be pre-trained models whose parameters do not need to be changed during the training process of the image processing model.

[0132] The above model training process, through joint training of the ranking sub-model and the diffusion sub-model, makes the trained ranking sub-model more robust.

[0133] In one implementation, based on the degree of matching between sample text features and sample image features, a ranking sub-model is used to determine the target transition image from the sample transition images, including:

[0134] From a set of multiple feature pairs containing text and image features, determine the target image features corresponding to the sample text features;

[0135] Based on the distance between the features of the target image and the features of the sample image, a ranking sub-model is used to determine the target transition image from the sample transition images.

[0136] Based on the difference between the sample image features and sample text features corresponding to the target transition image, a second loss is constructed, including:

[0137] A second loss is constructed based on the distance between the sample image features corresponding to the target transition image and the target image features.

[0138] The target image feature can be an image feature in a preset plurality of text feature and image feature pairs having a sample text feature. Specifically, a plurality of text feature and image feature pairs can be preset, and after the sample text feature is determined, a text feature and image feature pair containing the sample text feature can be determined from the text feature and image feature pairs, and the image feature in the text feature and image feature pair is taken as the target image feature. The preset text feature and image feature pair can be obtained by contrastive training on a large amount of image-text pair data, mapping the image and the text into a shared embedding space, so that the relevant image and the text are closer in the embedding space, and the irrelevant image and the text are farther in the embedding space. After the contrastive training is completed, the distance between the relevant image and the text in the embedding space should be smaller than the distance between the irrelevant image and the text in the embedding space. At this time, the preset text feature and image feature pair can be obtained.

[0139] In an example, the preset text feature and image feature pair can be obtained by a trained contrastive language-image pre-training (CLIP) model, and the image encoding module and the text encoding module can be a text encoder and an image encoder of the contrastive language-image pre-training model, respectively. Specifically, the text information output by the language model can be encoded by using the text encoder of the contrastive language-image pre-training model to obtain the sample text feature, the sample transition image output by the diffusion sub-model can be encoded by using the image encoder of the contrastive language-image pre-training model to obtain the sample image feature, and then the target image feature corresponding to the sample text feature can be determined according to the text feature and image feature pair determined by the contrastive language-image pre-training model.

[0140] Further, the feature distances between the sample image features of the plurality of sample transition images corresponding to the same set of first image and second image and the target image feature can be calculated, and the feature distances can be sorted by using the ranking sub-model, and the target transition image can be selected from the plurality of sample transition images, and the feature distance corresponding to the target transition image is the smallest among the feature distances. After the target transition image is determined, the second loss corresponding to the ranking sub-model can be determined according to the feature distance corresponding to the target transition image.

[0141] In the embodiments of the present specification, the sample image feature of the sample transition image output by the diffusion sub-model is matched with the target image feature to determine the second loss, and then the effect of the transition image output by the image processing process can be improved by combining the first loss and the second loss.

[0142] It should be noted that the image processing method provided in the embodiments of the present specification can be executed by an image processing device, or a control module in the image processing device for executing the image processing method. In the embodiments of the present specification, the image processing device is taken as an example to illustrate the image processing device provided in the embodiments of the present specification.

[0143] According to the image processing method described above, based on the same technical concept, one or more embodiments of the present application also provide an image processing device. Figure 6 A module composition schematic diagram of the image processing device provided in one or more embodiments of the present application is shown in the figure. Figure 6 As shown in the figure, the image processing device 600 includes:

[0144] The encoding module 610 is configured to perform first encoding processing on the first image and the second image to obtain first image features and second image features.

[0145] The interpolation module 620 is configured to perform interpolation processing on the first image features and the second image features to obtain first interpolation features.

[0146] The decoding module 630 is configured to perform decoding processing on the first interpolation features to obtain a transition image and insert the transition image between the first image and the second image.

[0147] The image processing device provided in the embodiments of the present application performs first encoding processing on the first image and the second image to be interpolated, and performs interpolation processing on the first image features and the second image features obtained by encoding to obtain first interpolation features, and then performs decoding processing on the first interpolation features to generate a transition image. This process converts the interpolation between images into the interpolation between image features. Compared with the original image, the information amount of the image features is greatly reduced, which effectively reduces the difficulty of implementing the image interpolation process and improves the accuracy of the generated transition image, realizes the interpolation operation of any two images, and further improves the robustness of the transition image generation.

[0148] It should be noted that the embodiments of the image processing device in the present application are based on the same inventive concept as the embodiments of the image processing method in the present application, and therefore the specific implementation of the embodiments can refer to the foregoing corresponding image processing method embodiments, and the repeated parts will not be described herein.

[0149] The above-mentioned modules in the image processing device can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the terminal device or the processor in the server in hardware form, or can be stored in the memory in the terminal device or the memory in the server in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0150] Further, based on the same technical conception as the above-described image processing method, one or more embodiments of the present application further provide an electronic device for executing the above-described image processing method, Figure 7 A structural schematic diagram of an electronic device provided by one or more embodiments of the present application is shown in FIG. 7.

[0151] Based on the same technical conception, one or more embodiments of the present application further provide an electronic device, as shown in FIG. 7. Figure 7 The electronic device can have great differences due to different configurations or performances, and can include one or more processors 701 and memories 702, and the memories 702 can store one or more application programs or data. The memories 702 can be temporary memories or persistent memories. The application programs stored in the memories 702 can include one or more modules (not shown in the figure), and each module can include a series of computer executable instructions in the electronic device. Further, the processor 701 can be configured to communicate with the memory 702 and execute the series of computer executable instructions in the memory 702 on the electronic device. The electronic device can further include one or more power supplies 703, one or more wired or wireless network interfaces 704, one or more input / output interfaces 705, and one or more keyboards 706.

[0152] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and the one or more programs can include one or more modules, and each module can include a series of computer executable instructions in the electronic device, and the configuration of the one or more processors to execute the one or more programs includes computer executable instructions for:

[0153] performing first encoding processing on the first image and the second image to obtain first image features and second image features;

[0154] performing interpolation processing on the first image features and the second image features to obtain first interpolation features;

[0155] performing decoding processing on the first interpolation features to obtain a transition image, and inserting the transition image between the first image and the second image.

[0156] The electronic device provided by one or more embodiments of the present application performs first encoding processing on a first image and a second image to be interpolated, performs interpolation processing on first image features and second image features obtained by encoding, obtains first interpolation features, and further performs decoding processing on the first interpolation features to generate a transition image. This process converts interpolation between images into interpolation between image features. Compared with original images, the amount of information of image features is greatly reduced, which effectively reduces the difficulty of implementing image interpolation and improves the accuracy of the generated transition image, realizes interpolation operation of any two images, and further improves the robustness of the generated transition image.

[0157] It should be noted that the embodiments of the electronic device in the present application are based on the same inventive concept as the embodiments of the image processing method in the present application, and therefore the specific implementation of the embodiments can be referred to the foregoing implementation of the corresponding image processing method, and the repeated parts will not be described again.

[0158] Further, corresponding to the image processing method described above, based on the same technical concept, one or more embodiments of the present application also provide a storage medium for storing computer executable instructions. In one specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc. The computer executable instructions stored in the storage medium can implement the following processes when executed by a processor:

[0159] performing first encoding processing on a first image and a second image to obtain first image features and second image features;

[0160] performing interpolation processing on the first image features and the second image features to obtain first interpolation features;

[0161] performing decoding processing on the first interpolation features to obtain a transition image, and inserting the transition image between the first image and the second image.

[0162] The computer executable instructions stored in the storage medium provided by one or more embodiments of the present application perform first encoding processing on a first image and a second image to be interpolated, perform interpolation processing on first image features and second image features obtained by encoding, obtain first interpolation features, and further perform decoding processing on the first interpolation features to generate a transition image. This process converts interpolation between images into interpolation between image features. Compared with original images, the amount of information of image features is greatly reduced, which effectively reduces the difficulty of implementing image interpolation and improves the accuracy of the generated transition image, realizes interpolation operation of any two images, and further improves the robustness of the generated transition image.

[0163] It should be noted that the embodiment of the storage medium in the present application is based on the same inventive concept as the image processing method in the present application, and therefore the specific implementation of this embodiment can be referred to the implementation of the corresponding image processing method described above, and the repeated parts will not be described herein.

[0164] Further, corresponding to the image processing method described above, based on the same technical concept, one or more embodiments of the present application also provide a computer program product, which comprises a computer program, and when the computer program is executed by a processor, the following process can be realized:

[0165] performing first encoding processing on the first image and the second image to obtain first image features and second image features;

[0166] performing interpolation processing on the first image features and the second image features to obtain first interpolation features;

[0167] performing decoding processing on the first interpolation features to obtain a transition image, and inserting the transition image between the first image and the second image.

[0168] The computer program in the computer program product provided by one or more embodiments of the present application, when executed by a processor, performs first encoding processing on the first image and the second image to be interpolated, and performs interpolation processing on the first image features and the second image features obtained by encoding to obtain first interpolation features, and then performs decoding processing on the first interpolation features to generate a transition image. This process converts the interpolation between images into the interpolation between image features, and the information amount of the image features is greatly reduced compared to the original image, which effectively reduces the difficulty of implementing the image interpolation process and improves the accuracy of the generated transition image, realizes the interpolation operation of any two images, and further improves the robustness of the transition image generation.

[0169] It should be noted that the embodiment of the computer program product in the present application is based on the same inventive concept as the embodiment of the image processing method in the present application, and therefore the specific implementation of this embodiment can be referred to the implementation of the corresponding image processing method described above, and the repeated parts will not be described herein.

[0170] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps in a claim can be performed in an order other than the order in which the acts or steps are recited, and still accomplish the desired result. In addition, the process depicted in the accompanying figures does not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0171] In the 1990s, it was relatively easy to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has evolved, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flows into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming the PLD, rather than by ordering a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented using "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0172] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can also be implemented to perform the same functions in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can even be considered as both a software module implementing a method and a structure within a hardware component.

[0173] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0174] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the embodiments of the present application.

[0175] Those skilled in the art will appreciate that one or more embodiments of the application can provide a method, a system or a computer program product. Accordingly, one or more embodiments of the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0176] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0177] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0178] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0179] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0180] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0181] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0182] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0183] One or more embodiments of the application can be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. One or more embodiments of the application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0184] Various embodiments in this application are described in progressive manner, and the same or similar parts between various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0185] The above merely provides the example of the present document and is not intended to limit the present document. For those skilled in the art, the present document can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present document shall be included in the scope of claims of the present document.

Claims

1. An image processing method, characterized by, The method comprises: performing first encoding processing on the first image and the second image to obtain first image features and second image features; performing interpolation processing on the first image features and the second image features to obtain first interpolation features; performing decoding processing on the first interpolation features to obtain a transition image, and inserting the transition image between the first image and the second image.

2. The method of claim 1, wherein, The interpolation processing on the first image features and the second image features to obtain first interpolation features comprises: adding noise processing to the first image features and the second image features to obtain first image features and second image features after adding noise; performing interpolation processing on the first image features and the second image features after adding noise to obtain the first interpolation features.

3. The method of claim 2, wherein, The decoding processing on the first interpolation features to obtain a transition image comprises: obtaining a decoding condition of the first interpolation features; based on the decoding condition, using a diffusion sub-model to perform decoding processing on the first interpolation features to obtain the transition image, wherein the decoding processing comprises noise removal processing.

4. The method of claim 3, wherein, The decoding processing on the first interpolation features to obtain a transition image based on the decoding condition using a diffusion sub-model comprises: performing second encoding processing on the first image and the second image to obtain third image features and fourth image features; performing interpolation processing on the third image features and the fourth image features to obtain second interpolation features; adjusting the first interpolation features based on the second interpolation features to obtain adjusted first interpolation features; based on the decoding condition, using the diffusion sub-model to perform decoding processing on the adjusted first interpolation features to obtain the transition image.

5. The method of claim 4, wherein, The second encoding processing comprises M encoding types; the second encoding processing on the first image and the second image to obtain third image features and fourth image features comprises: based on the M encoding types, performing second encoding processing on the first image and the second image to obtain M third image features and M fourth image features; The interpolation processing on the third image features and the fourth image features to obtain second interpolation features comprises: performing interpolation processing on third image features and fourth image features belonging to the same encoding type to obtain M second interpolation features.

6. The method of claim 3, wherein, The decoding condition comprises text information obtained from the first image and the second image, and the process of obtaining the text information comprises the following steps: performing text description on the first image and the second image to obtain first image text and second image text; inputting the first image text and the second image text into a language model to obtain the text information.

7. The method of claim 3, wherein, The training process of the diffusion sub-model comprises the following steps: performing first encoding processing on a first sample image and a second sample image to obtain first sample image features and second sample image features; interpolating the first sample image feature and the second sample image feature to obtain a first sample interpolation feature after adding the first noise to the first sample image feature and the second sample image feature; based on the obtained decoding condition, using the diffusion sub-model to perform decoding processing on the first sample interpolation feature to obtain a sample transition image, and determining second noise removed by the decoding processing corresponding to the sample transition image; constructing a first loss according to a gap between the first noise and the second noise, and training the diffusion sub-model according to the first loss to obtain a trained diffusion sub-model.

8. The method of claim 7, wherein, A plurality of sample transition images correspond to the same set of first sample images and second sample images, and the training of the diffusion sub-model according to the first loss to obtain the trained diffusion sub-model includes: respectively encoding the decoding condition and the sample transition image corresponding to the same set of first sample images and second sample images to obtain a sample text feature of the decoding condition and a sample image feature of the sample transition image; based on a matching degree of the sample text feature and the sample image feature, using a ranking sub-model to determine a target transition image from the sample transition image; constructing a second loss according to a gap between the sample image feature corresponding to the target transition image and the sample text feature; training the diffusion sub-model and the ranking sub-model jointly according to the first loss and the second loss to obtain the trained diffusion sub-model and the ranking sub-model.

9. An image processing apparatus characterized by comprising: including: an encoding module configured to perform first encoding processing on a first image and a second image to obtain a first image feature and a second image feature; an interpolation module configured to interpolate the first image feature and the second image feature to obtain a first interpolation feature; a decoding module configured to perform decoding processing on the first interpolation feature to obtain a transition image, and insert the transition image between the first image and the second image.

10. An electronic device, comprising: including: a processor; and a memory arranged to store computer executable instructions configured to be executed by the processor, the executable instructions including instructions for performing steps in the image processing method according to any one of claims 1-8.

11. A storage medium, characterized by The storage medium is configured to store computer executable instructions, and the executable instructions enable a computer to perform the image processing method according to any one of claims 1-8.

12. A computer program product, characterised in that, The computer program is configured to be executed by a processor to implement the image processing method according to any one of claims 1-8.

Images