Image processing method and apparatus

Abstract

This specification provides an image processing method and apparatus. The image processing method includes: acquiring image editing information, an original image, and a reference image; encoding the original image and the reference image using the encoding module of an image generation model to obtain original image encoding features and reference image encoding features; constructing inverse velocity field features based on the original image encoding features and image editing information; constructing forward velocity field features based on the reference image encoding features, image editing information, and image features corresponding to the previous feature processing cycle; determining image features corresponding to the current feature processing cycle based on the inverse velocity field features, forward velocity field features, and semantic alignment velocity field features corresponding to the original image; and decoding the image features corresponding to the target feature processing cycle using the decoding module of the image generation model until the target feature processing cycle meets the image generation conditions to obtain the target image corresponding to the image editing information.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of image processing technology, and particularly to image processing methods and apparatus. Background Technology

[0002] Artificial intelligence-based image editing technology has developed rapidly in recent years and has been widely used in fields such as post-processing of photography, product design, and e-commerce. Traditional image editing models mainly rely on text instructions to modify input images. However, text descriptions have inherent limitations and cannot accurately express the complex needs of users for image details, resulting in the generated results failing to accurately capture the expected detailed features. To improve editing accuracy, multi-image editing technology introduces visual reference images as context, combining them with text instructions to process the target image. This method provides intuitive visual information through reference images, enabling a more detailed description of the generated image's content, meeting the urgent need for high-precision image editing in scenarios such as e-commerce. However, in practical applications, when the original image and the reference image differ significantly in content structure, style features, or semantic information, existing technologies struggle to effectively align and fuse their key features, often resulting in problems such as detail distortion, feature inconsistency, or unstable generation processes. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, an image processing method is provided, comprising: Image editing information, original image, and reference image are acquired, and the encoding module of the image generation model is used to encode the original image and the reference image respectively to obtain the encoding features of the original image and the encoding features of the reference image. Construct the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and construct the forward velocity field feature corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information and the image features corresponding to the previous feature processing cycle; Based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image, the image features corresponding to the current feature processing cycle are determined; Until the target feature processing cycle meets the image generation conditions, the decoding module of the image generation model is used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information.

[0005] According to a second aspect of the embodiments of this specification, an image processing apparatus is provided, comprising: The encoding module is configured to acquire image editing information, the original image, and the reference image, and to encode the original image and the reference image using the encoding module of the image generation model, respectively, to obtain the original image encoding features and the reference image encoding features. The construction module is configured to construct the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and to construct the forward velocity field feature corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information and the image features corresponding to the previous feature processing cycle; The determination module is configured to determine the image features corresponding to the current feature processing cycle based on the reverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image. The decoding module is configured to decode the image features corresponding to the target feature processing cycle using the decoding module of the image generation model until the target feature processing cycle meets the image generation conditions, thereby obtaining the target image corresponding to the image editing information.

[0006] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the image processing method described above.

[0008] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the image processing method described above.

[0009] The image processing method provided in this embodiment, in order to improve the image editing effect, can acquire image editing information, original image, and reference image, and use the encoding module of the image generation model to encode the original image and the reference image respectively to obtain the original image encoding features and the reference image encoding features. Then, based on the original image encoding features and image editing information, the inverse velocity field features corresponding to the current feature processing cycle can be constructed, and based on the reference image encoding features, image editing information, and the image features corresponding to the previous feature processing cycle, the forward velocity field features corresponding to the current feature processing cycle can be constructed. Thus, based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image, the image features corresponding to the current feature processing cycle can be determined. This is then used to process the next feature processing cycle until the target feature processing cycle meets the image generation conditions. At this point, the decoding module of the image generation model can be used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information. This method achieves precise feature fusion and image generation by acquiring image editing information, original image, and reference image, encoding features, constructing inverse and forward velocity field features, combining semantic alignment to determine image features, and decoding to generate target images. It effectively solves the problem of feature alignment in multi-image editing, significantly improves the accuracy and robustness of image editing, and reduces detail distortion. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an image processing method provided in one embodiment of this specification; Figure 2a This is a schematic diagram of the model structure in an image processing method provided in one embodiment of this specification; Figure 2b This is a schematic diagram illustrating the image editing effect in an image processing method provided in one embodiment of this specification; Figure 3 This is an interactive schematic diagram of an image processing method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0011] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0012] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0014] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0015] The technical solutions provided in this application can be implemented using deep learning models. However, this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based language models (LM) or multimodal models (MM). First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0016] Rectified Flow (RF) is a commonly used generative model. The goal of generative models is to learn the distribution of data so as to generate new, real-world-like samples. Rectified Flow models the data generation process through specific flow mechanisms. It may utilize reversible transformations to convert simple initial distributions (such as Gaussian distributions) into complex data distributions, thereby achieving efficient sample generation. It has wide applications in tasks such as image generation.

[0017] FlowEdit is a text-based image editing algorithm that requires no training or diffusion inversion. Traditional image editing methods may require large amounts of training data and complex training processes, but FlowEdit breaks this limitation. It leverages the characteristics of generative models (such as Rectified Flow) and combines them with text commands to guide image editing operations. Users can provide natural language text commands for the algorithm to edit the input image accordingly, such as changing the color or shape of objects, adding or deleting objects, without requiring pre-training for the editing task or the complex step of diffusion inversion.

[0018] DPS (Diffusion Posterior Sampling) is an algorithm that enhances the consistency between generated and input images. Maintaining the correlation and consistency between the generated image and the input image is crucial in image generation and editing tasks. The DPS algorithm samples from the posterior distribution of a diffusion model, allowing the generated image to better preserve key features and information from the input image while adhering to the data distribution learned by the model. This improves the quality of the generated image and its matching degree with the input image.

[0019] Concept Attention is an algorithm for semantic segmentation based on the Rectified Flow (RF) model that requires no additional training. Semantic segmentation is the task of assigning each pixel in an image to a specific semantic category (such as people, vehicles, sky, etc.). The Concept Attention algorithm leverages the structure and characteristics of the Rectified Flow model to perform semantic segmentation on input images without additional training. It can identify regions corresponding to different semantic concepts by analyzing the model's internal feature representations, thus achieving semantic understanding of the image.

[0020] LPIPS (Learned Perceptual Image Patch Similarity) is a metric that measures the similarity between two images. Unlike traditional pixel-based similarity metrics (such as mean squared error), LPIPS considers the characteristics of human visual perception. It assesses similarity by calculating the differences between image patches in the feature space of a deep neural network, thus better reflecting human subjective perception of image similarity. The smaller the LPIPS value, the more similar the two images are perceived to be.

[0021] DINO Similarity is a metric that measures the similarity between specific objects in two images. DINO (Distilled Knowledge from Vision Transformers) is a self-supervised learning method based on the Visual Transformer. DINO Similarity utilizes feature representations learned by the DINO model to focus on specific objects in an image and calculate the similarity between them. This metric can more accurately assess the similarity of objects in appearance, pose, etc., and has important applications in tasks such as object recognition and image retrieval.

[0022] To address the aforementioned technical problems, this specification provides an image processing method. This specification also relates to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0023] In text-based image editing, the inherent ambiguity of natural language descriptions prevents users from precisely expressing their needs for modifying image details, leading to semantic ambiguity in the feature space mapping process of the image generation model. This imprecise correspondence between text instructions and image details makes it difficult for the model to accurately capture key visual elements in the user's intent, resulting in local deviations from the expected goals in the editing results. This problem directly weakens the detail control capability and output consistency of the image editing system, affecting the reliability of the system in key performance indicators, especially in applications requiring high-precision modifications. For example, in the product image processing process on an e-commerce platform, a user needs to adjust the color of a specific area in the product image from standard blue to a light green with subtle hue differences. When the user inputs the text instruction "change the product area to light green," the image generation model cannot distinguish the specific hue and saturation requirements in the user's intent because "light green" lacks a clear color parameter definition. In this scenario, the model extracts features solely based on the text description, resulting in the product area in the generated image exhibiting various non-target hue variations of green, such as yellowish or grayish greens, failing to meet the user's stringent requirements for detail precision. This issue necessitates repeated adjustments to instructions during the editing process, increasing operational complexity and reducing image processing efficiency. If these technical problems are not resolved, the deficiencies in detail control within image editing systems will persist, leading to increased editing failure rates and decreased user trust. In service sectors reliant on precise visual expression, such as product design and advertising production, this deficiency will limit the technology's applicability and may trigger a chain reaction of errors in subsequent processing stages, affecting the stability of the overall workflow and output quality. Therefore, a technical solution that can effectively improve the precision of detail control is urgently needed.

[0024] The image processing method provided in this embodiment, in order to improve the image editing effect, can acquire image editing information, original image, and reference image, and use the encoding module of the image generation model to encode the original image and the reference image respectively to obtain the original image encoding features and the reference image encoding features. Then, based on the original image encoding features and image editing information, the inverse velocity field features corresponding to the current feature processing cycle can be constructed, and based on the reference image encoding features, image editing information, and the image features corresponding to the previous feature processing cycle, the forward velocity field features corresponding to the current feature processing cycle can be constructed. Thus, based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image, the image features corresponding to the current feature processing cycle can be determined. This is then used to process the next feature processing cycle until the target feature processing cycle meets the image generation conditions. At this point, the decoding module of the image generation model can be used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information. This method achieves precise feature fusion and image generation by acquiring image editing information, original image, and reference image, encoding features, constructing inverse and forward velocity field features, combining semantic alignment to determine image features, and decoding to generate target images. It effectively solves the problem of feature alignment in multi-image editing, significantly improves the accuracy and robustness of image editing, and reduces detail distortion.

[0025] See Figure 1 , Figure 1 A flowchart of an image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0026] Step S102: Obtain image editing information, original image and reference image, and use the encoding module of the image generation model to encode the original image and the reference image respectively to obtain the original image encoding features and the reference image encoding features.

[0027] Step S104: Construct the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and construct the forward velocity field feature corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information and the image features corresponding to the previous feature processing cycle.

[0028] Step S106: Determine the image features corresponding to the current feature processing cycle based on the reverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image.

[0029] Step S108: Until the target feature processing cycle meets the image generation conditions, the decoding module of the image generation model is used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information.

[0030] The image processing method provided in this embodiment can be applied to any scenario, such as a virtual try-on scenario. Based on the clothing image, the person image, and editing instructions uploaded by the user, it can automatically generate an image of the user wearing specific clothing to assist the user in purchasing goods. It can also be applied to a promotional poster generation scenario. Based on the captured image uploaded by the user, a reference poster, and the poster generation requirements, it can automatically generate a poster image that matches the poster generation requirements.

[0031] In practice, the image processing method provided in this embodiment can be applied to any scenario with image generation requirements, and will not be elaborated further in this embodiment.

[0032] Specifically, image editing information refers to instructions or data used to guide the image editing process, which may include text descriptions, user interaction inputs, preset editing parameters, or style requirements. This information defines what modifications to the original image and how to use a reference image for guidance. The original image is the initial image provided to the image processing system as input before the editing operation begins. It is the basis for the editing operation, and its content will be modified or transformed. The reference image is an auxiliary image that provides visual context or style reference during the image editing process. It usually contains specific details, textures, or overall styles that the user wants to embody in the target image. An image generation model is a deep learning-based computational model that is trained to generate new images from input information (such as text, image features). This model typically includes components such as encoders and decoders, capable of learning the latent representation of an image and converting it into a visual image. The image generation model in this embodiment can use a trained image editing model that performs multi-image editing based on text instructions. It belongs to a type of RF (Rectified Flow) model.

[0033] Correspondingly, the encoding module is part of the image generation model. Its function is to convert the input image (such as the original image or reference image) into a low-dimensional feature representation with semantic information, i.e., encoded features. These encoded features capture the key content and structural information of the image. The original image encoded features are abstract representations obtained after processing the original image by the encoding module. They contain the visual content and structural information of the original image, used for subsequent feature processing and editing. The reference image encoded features are abstract representations obtained after processing the reference image by the encoding module. They contain the visual context and style information of the reference image, used to guide the generation of the target image. The current feature processing cycle refers to a time step or stage in the iterative feature processing during image generation. Image generation is typically a progressively refined process, with image features updated in each cycle. The inverse velocity field feature is a representation constructed during image generation based on the original image encoded features and image editing information. It can be understood as a trend of reverse diffusion or transformation from the current state to the original image state, used to guide the updating of image features. Forward velocity field features are a representation constructed during image generation based on the coded features of the reference image, image editing information, and image features from the previous feature processing cycle. They can be understood as a trend of forward diffusion or transformation from the current state to the reference image state, used to guide the updating of image features. The image features from the previous feature processing cycle refer to the image features obtained through iterative processing before the current feature processing cycle. They serve as the input basis for feature updates in the current cycle.

[0034] Correspondingly, semantic alignment velocity field features are a representation obtained through alignment processing based on the semantic information of the original image. Its purpose is to ensure that the semantic structure of the target image remains consistent with the original image during image editing, avoiding unnecessary semantic drift. The target feature processing cycle refers to the final feature processing cycle in the image generation process. When this cycle is reached, the image features are considered sufficiently complete and can be decoded to generate the final image. Image generation conditions refer to the criteria for determining whether the image generation process can end and the final image decoding can proceed. These can be based on time step, convergence, or a preset number of iterations. The decoding module is part of the image generation model; its function is to convert the image features, updated iteratively through multiple feature processing cycles, back into a visual image form, i.e., to generate the target image. The target image refers to the final output image after editing and generation using image processing methods. It integrates the content of the original image, the style of the reference image, and the requirements of image editing information.

[0035] Based on this, the method includes acquiring image editing information, an original image, and a reference image. Image editing information can be provided by the user through text input or a graphical interface; for example, the user can input "change the sky in the original image to blue and give it the cloud style of the reference image." The original image is the image to be edited, such as a landscape photograph. The reference image provides visual reference for editing, such as a sky photograph with specific cloud shapes and colors. After acquiring this information, the encoding module of the image generation model encodes the original image and the reference image separately to obtain original image encoding features and reference image encoding features. The encoding module can convert the image into a series of numerical vectors, which represent high-level semantic information such as color, texture, and shape. For example, the encoding features of the original image can capture the structure of its mountains and rivers, while the encoding features of the reference image can capture its unique cloud distribution and lighting effects.

[0036] Furthermore, a reverse velocity field feature corresponding to the current feature processing cycle is constructed based on the original image's encoded features and the image editing information. The reverse velocity field feature can be understood as a "correction direction," indicating how to adjust the current image features towards the feature space of the original image. For example, the original image's encoded features can be linearly combined with the image editing information, or processed using a small neural network to generate a preliminary reverse velocity field. This reverse velocity field feature aims to ensure that the generated image does not completely deviate from the basic structure of the original image during editing. Simultaneously, a forward velocity field feature corresponding to the current feature processing cycle is constructed based on the reference image's encoded features, the image editing information, and the image features corresponding to the previous feature processing cycle. The forward velocity field feature indicates how to adjust the current image features towards the feature space of the reference image and the direction desired by the editing information. For example, the reference image's encoded features, the image features corresponding to the previous feature processing cycle, and the image editing information can be weighted and averaged, or a lookup table can be used to determine the forward velocity field. This forward velocity field feature aims to incorporate the style and editing instructions of the reference image into the generated image.

[0037] Furthermore, based on the inverse velocity field feature, the forward velocity field feature, and the semantically aligned velocity field feature corresponding to the original image, the image features corresponding to the current feature processing cycle are determined. This step is the core of the image feature iterative update. The inverse velocity field feature provides a trend for preserving the content of the original image, the forward velocity field feature provides a trend for fusing the style of the reference image and editing instructions, and the semantically aligned velocity field feature ensures the consistency of image semantics during the editing process. For example, these three velocity field features can be simply summed or weighted to obtain a comprehensive velocity field, which is then applied to the image features corresponding to the previous feature processing cycle to update the image features corresponding to the current feature processing cycle. This update process can be a simple feature overlay or a more complex feature fusion operation.

[0038] Finally, once the target feature processing cycle meets the image generation conditions, the decoding module of the image generation model decodes the image features corresponding to that target feature processing cycle to obtain the target image corresponding to the image editing information. The image generation conditions can be a preset number of iterations, for example, after t feature processing cycles, the image features are considered sufficiently stable. When this condition is met, the iteration process stops, and the final image features are sent to the decoding module. The decoding module converts these abstract feature representations back into visible pixel data, thereby generating the target image desired by the user. For example, the decoding module can be a deconvolutional network that progressively upsamples the feature map and converts it into a high-resolution image.

[0039] For example, suppose user A wants to edit an original image showing a clear sky and a mountain peak. User A wants to change the sky's color from blue to orange and give it the unique texture and lighting effects of sunset clouds in a reference image. User A inputs the text command "change the sky to orange and give it sunset cloud texture" and provides the original image and reference image as image editing information. First, the image processing method obtains the image editing information, original image, and reference image provided by user A. Then, the encoding module of the image generation model processes the original image and reference image, extracting the original image encoding features and the reference image encoding features, respectively. The original image encoding features capture the outline of the mountain peak and the regional information of the original sky, while the reference image encoding features contain the color gradation, texture details, and lighting direction of the sunset clouds.

[0040] Furthermore, in the first feature processing cycle, the system constructs an inverse velocity field feature based on the original image encoding features and image editing information ("turn the sky orange"). This inverse velocity field feature indicates that during editing, the sky region should maintain its original position and approximate shape, avoiding unnecessary modification to the mountain region. Simultaneously, a forward velocity field feature is constructed based on the reference image encoding features, image editing information, and initial image features (e.g., a transformation of the original image encoding features). This forward velocity field feature guides the sky region to evolve towards an orange hue and sunset cloud texture. In addition, the system generates a semantically aligned velocity field feature based on the semantic information of the original image (e.g., identifying the sky region through a pre-trained model) to ensure that the semantic boundaries of the sky region are not blurred or confused with the mountain region in subsequent feature updates. Subsequently, the system fuses these three velocity field features and applies them to the initial image features to determine the image features corresponding to the first feature processing cycle. This image feature initially displays an orange sky and some blurred cloud textures, but the overall effect is still not refined. This process continues iteratively. In each subsequent feature processing cycle, the system uses the updated image features of the current cycle as the "image features corresponding to the previous feature processing cycle" and reconstructs new inverse velocity field features and forward velocity field features. The inverse velocity field features continuously constrain the image features to prevent them from deviating from the structure of the original image, such as ensuring the shape of the mountains remains unchanged. The forward velocity field features continuously guide the image features towards the editing goals of the sunset cloud style and orange sky of the reference image, such as gradually refining the edges and lighting effects of the clouds. The semantic alignment velocity field features continuously maintain the semantic boundary clarity between the sky and mountain regions throughout the iteration process. Through multiple feature processing cycles, the image features are gradually refined. For example, in the 50th cycle, the orange color of the sky is very obvious, and the texture of the clouds begins to become clear. In the 100th cycle, the details of the clouds, the lighting effects, and the connection with the mountains all reach the level expected by user A. Finally, when the preset target feature processing cycle (e.g., the 100th cycle) is reached and the image generation conditions are met, the iteration process stops. The image features obtained at this point are sent to the decoding module of the image generation model. The decoding module converts these highly abstract features into visual pixel data, generating a new target image. In this target image, the sky displays a beautiful orange sunset, the clouds possess the unique texture and lighting of the reference image, while the mountain peaks retain the shape of the original image, meeting User A's editing needs. This example demonstrates that precise control over image content and style can be achieved through multi-velocity field fusion and iterative updates.

[0041] In summary, to improve image editing results, image editing information, the original image, and a reference image can be acquired. The encoding module of the image generation model is then used to encode the original image and the reference image respectively, obtaining the original image encoding features and the reference image encoding features. Subsequently, the inverse velocity field features corresponding to the current feature processing cycle can be constructed based on the original image encoding features and image editing information, and the forward velocity field features corresponding to the current feature processing cycle can be constructed based on the reference image encoding features, image editing information, and the image features corresponding to the previous feature processing cycle. This allows for the determination of the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image. This process is repeated for the next feature processing cycle until the target feature processing cycle meets the image generation conditions. Finally, the decoding module of the image generation model is used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information. This method achieves precise feature fusion and image generation by acquiring image editing information, original image, and reference image, encoding features, constructing inverse and forward velocity field features, combining semantic alignment to determine image features, and decoding to generate target images. It effectively solves the problem of feature alignment in multi-image editing, significantly improves the accuracy and robustness of image editing, and reduces detail distortion.

[0042] When directly constructing inverse velocity field features, the robustness of the results may be insufficient due to a single feature mapping or processing path, making them susceptible to noise or local information bias, thus affecting the accuracy of the final image features. Therefore, in this embodiment, constructing the inverse velocity field features corresponding to the current feature processing cycle based on the original image encoding features and the image editing information includes: Based on the original image encoding features, construct multiple first image mapping features corresponding to the current feature processing cycle; based on the multiple first image mapping features and the editing information corresponding to the original image in the image editing information, construct multiple initial reverse velocity field features; determine the reverse velocity field features corresponding to the current feature processing cycle based on the multiple initial reverse velocity field features.

[0043] Specifically, multiple first-image mapping features are a series of intermediate representations generated based on the original image's encoded features, designed to capture the latent information of the original image from different angles or through different transformations. These can serve as diverse inputs for subsequent processing to enhance the understanding and utilization of the original image features. For example, these mapping features can be generated by applying different nonlinear activation functions, different sets of convolutional kernels, or introducing small random perturbations into the feature space to the original image's encoded features. Furthermore, diverse mapping features can also be obtained by inputting the original image's encoded features into multiple parallel sub-networks with slightly different parameters.

[0044] Multiple initial inverse velocity field features are independently predicted inverse velocity fields based on each first image mapping feature and image editing information. Each initial inverse velocity field feature represents a prediction of back-diffusion of image features in conjunction with editing information from a specific mapping perspective. For example, each first image mapping feature can be concatenated with image editing information and then processed in a shared or independent prediction network (e.g., a multilayer perceptron or a small convolutional network) to generate a corresponding initial inverse velocity field feature for each mapping feature. Alternatively, different first image mapping features and image editing information can be iteratively input into the same processing module, generating an initial inverse velocity field feature with each processing step.

[0045] In determining the reverse velocity field features based on multiple initial reverse velocity field features, an aggregation calculation method can be used. This aggregation calculation includes, but is not limited to, arithmetic mean calculation, weighted average calculation, median calculation, geometric mean calculation, and harmonic mean calculation, aiming to integrate different prediction results and improve the stability and accuracy of the final reverse velocity field features. For example, a simple arithmetic mean can be used, which involves summing the corresponding elements of all initial reverse velocity field features and then dividing by the number of features. Alternatively, a weighted average can be used, where the weights can be dynamically determined based on the confidence level of each initial reverse velocity field feature or through a learning mechanism. In specific implementation, the method for determining the reverse velocity field features can be selected according to actual needs, and this embodiment does not impose any limitations.

[0046] Based on this, multiple first image mapping features corresponding to the current feature processing cycle are first constructed according to the original image encoding features, thus providing diverse input perspectives for the construction of inverse velocity field features. Subsequently, based on these diverse first image mapping features and the editing information corresponding to the original image in the image editing information, multiple initial inverse velocity field features are constructed respectively, enabling the system to obtain inverse velocity field information from multiple independent prediction paths. Finally, by aggregating these multiple initial inverse velocity field features and fusing the results of different prediction paths, the inverse velocity field features corresponding to the current feature processing cycle are obtained. This multi-path construction and average aggregation mechanism effectively avoids the bias and instability that may be caused by single prediction, significantly enhancing the robustness and accuracy of the inverse velocity field features. In this way, based on the encoding module of the image generation model encoding the original image and obtaining the original image encoding features, the inverse velocity field features can be constructed more stably and accurately, laying a solid foundation for subsequently determining the image features corresponding to the current feature processing cycle.

[0047] For example, when constructing the inverse velocity field feature corresponding to the current feature processing cycle, ten different first image mapping features can be derived from the original image coding features. This can be achieved by applying different random masks to the original image coding features or by using different feature extraction layers. Next, for each first image mapping feature, it is input along with the editing information of the corresponding original image from the image editing information into the inverse velocity field processing module of the image generation model to independently generate an initial inverse velocity field feature. For example, if ten first image mapping features are generated, ten initial inverse velocity field features will be generated accordingly. Finally, these ten initial inverse velocity field features are averaged element-wise to obtain the final inverse velocity field feature corresponding to the current feature processing cycle.

[0048] By employing the aforementioned technical solutions and strategies of multi-path feature mapping and average aggregation, the robustness and accuracy of inverse velocity field feature construction are effectively improved. This helps reduce the noise and uncertainty that may be introduced by a single prediction, making the inverse diffusion direction in the image generation process more stable and reliable. Consequently, higher quality image output that better meets the editing intent can be obtained in subsequent image feature determination and target image generation stages. When constructing inverse velocity field features, the key to achieving high-quality image editing lies in how to generate refined intermediate features for subsequent processing and ensure the accuracy and robustness of the inverse velocity field features. Therefore, in this embodiment, the construction of any one of the plurality of first image mapping features includes: Determine the Gaussian noise parameters corresponding to the current feature processing cycle, and construct the first image mapping feature corresponding to the current feature processing cycle based on the Gaussian noise parameters and the original image encoding features; wherein, the construction of any one of the plurality of initial inverse velocity field features includes: processing the first image mapping feature and the editing information corresponding to the original image in the image editing information using the inverse velocity field processing module of the image generation model to obtain the initial inverse velocity field feature.

[0049] Specifically, Gaussian noise parameters refer to the statistics used to generate random noise that conforms to a Gaussian distribution, typically including the mean and standard deviation. These parameters can be preset to fixed values, such as a mean of 0 and a standard deviation of 1, and used directly in each feature processing cycle; alternatively, they can be dynamically adjusted according to the current feature processing cycle (e.g., time step or number of iterations), for example, gradually decreasing the standard deviation as the cycle progresses to simulate the gradual reduction of noise; or, appropriate Gaussian noise parameters can be predicted or selected using a small neural network or lookup table based on the characteristics of the original image's encoded features or the complexity of the image editing information. The first image mapping feature is an intermediate feature representation constructed by combining the Gaussian noise parameters and the original image's encoded features, introducing randomness or perturbation into the encoded information of the original image. This feature can be constructed by performing element-wise addition or multiplication operations between the Gaussian noise and the original image's encoded features; it can also be generated by using a small neural network with the Gaussian noise parameters and the original image's encoded features as input; or more complex fusion mechanisms, such as attention mechanisms, can be used to weight the original image's encoded features according to the importance of the Gaussian noise parameters, thereby generating the first image mapping feature. The inverse velocity field processing module of the image generation model is a dedicated component that predicts an inverse velocity field based on input features. This module can be a U-Net-based neural network, adept at processing image features and outputting a velocity field with a similar dimensionality to the input feature space; it can also be a Transformer-based architecture, using self-attention to capture long-distance dependencies between features to predict the velocity field; or it can be a multilayer perceptron network, suitable for processing flattened feature vectors and outputting the corresponding velocity field. The image editing information corresponding to the original image is a subset of the image editing information, specifically targeting editing instructions or objectives for the original image. This information can be text descriptions, such as "make the sky blue"; semantic masks indicating the areas in the image to be edited and their target semantic categories; style reference images providing visual information about the target style; or even user-interacted strokes or doodles indicating the editing area and direction. The initial inverse velocity field features are the direct output of the inverse velocity field processing module, representing a preliminary estimate of denoising or editing from noisy or perturbed image features towards the original image in the current feature processing cycle. The feature can be a tensor with the same spatial resolution and number of channels as the image feature, with each pixel or feature point containing a velocity vector; it can also be a vector in the latent space, representing the direction and magnitude of movement of the latent feature; or it can be a discrete displacement field, indicating the target position that each pixel or feature point should move to.

[0050] Based on this, when constructing any one of multiple first image mapping features, the Gaussian noise parameter corresponding to the current feature processing cycle is first determined, and then the first image mapping feature is constructed based on this Gaussian noise parameter and the original image encoding features. This approach introduces Gaussian noise, giving each first image mapping feature a certain degree of random perturbation based on the original image encoding features, thereby increasing the diversity and robustness of subsequent processing. Next, when constructing any one of multiple initial inverse velocity field features, the inverse velocity field processing module of the image generation model is used to process the constructed first image mapping feature and the corresponding editing information of the original image in the image editing information as input, thereby obtaining the initial inverse velocity field feature. The inverse velocity field processing module, as a specially designed component, can effectively transform noisy image features and editing instructions into a velocity field that guides the direction of denoising or editing. Through this mechanism, this application can provide a series of initial inverse velocity field features with subtle differences but all pointing to the direction of the original image for subsequent aggregation calculations. This results in a more stable, accurate, and somewhat random inverse velocity field feature after averaging, which can cope with complex editing scenarios, thereby improving the flexibility and generation quality of image editing.

[0051] For example, when constructing the first image mapping feature, the Gaussian noise parameter can be determined based on the time step of the current feature processing cycle. For instance, in a diffusion model, a smaller time step usually means a lower noise level; therefore, a standard deviation inversely proportional to the time step can be set. Then, the Gaussian noise generated by this parameter is directly superimposed on the original image encoding feature to form the first image mapping feature. For example, if the original image encoding feature is a tensor of dimension C×H×W, Gaussian noise can also generate a tensor of the same dimension, which can be combined with the original image encoding feature through element-wise addition. When constructing the initial inverse velocity field feature, the inverse velocity field processing module of the image generation model can be a neural network based on a U-Net structure. This U-Net network receives the first image mapping feature and the editing information corresponding to the original image in the image editing information as input. The image editing information can be encoded into a conditional vector or feature map matching the dimension of the feature map space, and then incorporated into the encoder or bottleneck layer of the U-Net through a cross-attention mechanism or feature concatenation. The U-Net network, through its encoder-decoder structure, learns to predict pixel-level inverse velocity fields from noisy features and editing instructions. These velocity fields indicate how each pixel should move in the latent space to achieve denoising and editing goals.

[0052] Through the aforementioned technical solution, this application introduces Gaussian noise parameters to generate the first image mapping feature when constructing the inverse velocity field feature. This allows the model to inject a moderate degree of randomness when processing the original image encoding features, thereby enhancing the model's robustness to input variations. Simultaneously, by utilizing a dedicated inverse velocity field processing module to process the first image mapping feature and editing information, the editing intent can be captured more accurately, and the effective inverse velocity field can be predicted. This method, combining noise perturbation and a dedicated processing module, makes the generated initial inverse velocity field features more diverse and targeted, resulting in a more stable and accurate final inverse velocity field feature obtained through aggregation calculation. This not only improves the flexibility of image editing and the quality of generated images but also enables the model to better handle complex editing instructions and varied image content, thereby achieving more refined and natural image generation effects during image editing. When constructing positive velocity field features, effectively fusing reference image coding features, image editing information, and image features from the previous feature processing cycle to ensure the accuracy and stability of the positive velocity field features, thereby guiding the precise evolution of image features, is a problem that needs to be solved. In this embodiment, the step of constructing the positive velocity field features corresponding to the current feature processing cycle based on the reference image coding features, the image editing information, and image features from the previous feature processing cycle includes: Based on the plurality of first image mapping features and the reference image encoding features, a plurality of second image mapping features corresponding to the current feature processing cycle are constructed; based on the plurality of second image mapping features, the image features corresponding to the previous feature processing cycle, and the editing information corresponding to the reference image in the image editing information, a plurality of initial positive velocity field features are constructed; based on the plurality of initial positive velocity field features, the positive velocity field features corresponding to the current feature processing cycle are determined.

[0053] Specifically, the multiple first image mapping features are constructed based on the original image coding features. They can be feature representations obtained by transforming or perturbing the original image coding features in different ways. For example, multiple first image mapping features can be generated by adding noise of different intensities or patterns to the original image coding features, or by applying different feature extractors or transformation functions. These features aim to capture the intrinsic information of the original image from different perspectives, providing a diverse foundation for subsequent feature fusion. The reference image coding features are obtained by encoding the reference image using the coding module of the image generation model. They contain the style, content, or semantic information of the reference image and are key to guiding the image generation process towards the reference image. The multiple second image mapping features are constructed by combining multiple first image mapping features of the original image and reference image coding features. They are intermediate feature representations that fuse information from both the original and reference images. These features aim to associate the characteristics of the original image with the style or content of the reference image, providing richer contextual information for the construction of positive velocity field features. The construction method can be through feature concatenation followed by convolution, or through a weighted fusion of the two types of features using an attention mechanism. The image features corresponding to the previous feature processing cycle are those determined in the previous iteration step. They represent the intermediate state of the current image generation process and are important inputs for calculating the forward velocity field features, used to capture the evolution of image features over time. The editing information corresponding to the reference image in the image editing information consists of user- or system-preset editing instructions or constraints related to the reference image, such as style transfer intensity and content retention regions. These guide the generation direction of the forward velocity field features, ensuring the final image meets editing requirements. The multiple initial forward velocity field features are multiple preliminary forward velocity field predictions generated based on the aforementioned multiple inputs (multiple second image mapping features, image features corresponding to the previous feature processing cycle, and reference image editing information). Generating multiple initial features increases the robustness of the predictions and provides diverse samples for subsequent aggregation calculations. For example, multiple lightweight neural network modules can be run in parallel, each receiving the same input but with different internal parameters or random initialization, thereby generating different initial forward velocity field features. Determining the forward velocity field features based on the multiple initial forward velocity field features means aggregating these preliminary predictions to obtain a more stable and reliable forward velocity field feature. In practice, this processing can be achieved through aggregation calculations, which include, but are not limited to, arithmetic mean calculations, weighted average calculations, median calculations, geometric mean calculations, and harmonic mean calculations. These methods can effectively reduce the noise and uncertainty of individual predictions and improve the accuracy of the final positive velocity field features. For example, a simple arithmetic mean can be used, or a weighted average can be calculated based on the confidence levels of each initial feature.In practice, the method for determining the characteristics of the positive velocity field can be selected according to actual needs, and this embodiment does not impose any limitations.

[0054] Based on this, multiple second image mapping features are first constructed using multiple first image mapping features and reference image encoding features, achieving deep fusion of original image information and reference image information, providing multi-angle and multi-level input for the construction of forward velocity field features. On this basis, multiple initial forward velocity field features are constructed based on these multiple second image mapping features, the image features corresponding to the previous feature processing cycle, and the editing information of the corresponding reference image in the image editing information. This multi-path feature construction method can capture multiple evolution trends that may exist during the forward diffusion process, thus avoiding the bias that may be caused by a single prediction. Subsequently, by aggregating and calculating these multiple initial forward velocity field features, the prediction results are effectively smoothed, randomness and uncertainty are reduced, thereby obtaining a more stable and accurate forward velocity field feature corresponding to the current feature processing cycle. This strategy of multiple predictions and average aggregation significantly enhances the robustness and accuracy of the forward velocity field features, enabling the iterative update process of image features to evolve more accurately towards the target image, thus solving the problems of complex information fusion and insufficient prediction stability in the construction of forward velocity field features.

[0055] For example, when constructing the positive velocity field features corresponding to the current feature processing cycle, three first image mapping features can be generated firstly based on the original image encoding features. These three features can be obtained by adding Gaussian noise with different standard deviations to the original image encoding features. Next, these three first image mapping features are concatenated with the reference image encoding features and processed by three independent convolutional neural network modules to construct three second image mapping features. Each convolutional neural network module can contain multiple convolutional layers, activation functions, and pooling layers to extract fused features at different levels. Then, these three second image mapping features, the image features corresponding to the previous feature processing cycle, and the editing information of the corresponding reference image in the image editing information (e.g., a vector representing style transfer intensity) are used as inputs and fed into three parallel positive velocity field prediction networks. Each prediction network can be a multilayer perceptron or a small U-Net structure, and each outputs an initial positive velocity field feature. Finally, element-wise aggregation calculations are performed on these three initial positive velocity field features to obtain the final positive velocity field features corresponding to the current feature processing cycle.

[0056] Through the above technical solution, this application can construct positive velocity field features more stably and accurately. This method of multiple feature mapping and multiple velocity field prediction and aggregation calculation effectively reduces the error and uncertainty of a single prediction, enabling image features to more accurately integrate the style and content of the reference image during the iteration process and respond to editing information, thereby generating a high-quality target image that meets expectations. Accurately constructing the second image mapping features and determining appropriate visual context features under different processing cycles (especially the first feature processing cycle) are crucial for ensuring the accuracy of the positive velocity field features and the coherence of the image generation process. If these features are not constructed precisely enough or lack context awareness, the generated positive velocity field features may fail to accurately reflect the target editing intent, thus affecting the quality and editing effect of the final target image. Therefore, in this embodiment, the construction of any one of the plurality of second image mapping features includes: A first image mapping feature corresponding to the current feature processing cycle is determined, and a second image mapping feature corresponding to the current feature processing cycle is constructed based on the first image mapping feature, the original image encoding feature, and the reference image encoding feature. The construction of any one of the plurality of initial positive velocity field features includes: determining a visual context feature based on the image feature corresponding to the previous feature processing cycle, wherein, when the current feature processing cycle is the first feature processing cycle, the visual context feature is determined based on the reference image; and using the positive velocity field processing module of the image generation model, processing the editing information corresponding to the reference image in the image editing information, the visual context feature, and the second image mapping feature to obtain the initial positive velocity field feature.

[0057] Specifically, determining the first image mapping feature corresponding to the current feature processing cycle refers to obtaining the image feature representation generated in the current iteration step by the combined effect of the original image encoding features and Gaussian noise parameters. This feature reflects the state of the original image at the current noise level and is the basis for subsequent feature fusion. This can be a multidimensional tensor, such as a feature map extracted from the original image encoding features through a convolutional neural network, whose dimensions may be H x W x C (height, width, number of channels).

[0058] Based on the first image mapping feature, the original image encoding feature, and the reference image encoding feature, a second image mapping feature corresponding to the current feature processing cycle is constructed. This aims to effectively fuse information from the original image under the current noise state, the global encoding information of the original image, and the global encoding information of the reference image. This ensures that the second image mapping feature simultaneously contains details and overall structure of the source image, as well as style or content information of the target reference image. For example, these three features can be concatenated along the channel dimension, and then fused and dimensionality reduced through one or more convolutional layers to generate a new second image mapping feature. Alternatively, an attention mechanism, such as cross-attention, can be used to enable the first image mapping feature to query and integrate relevant information from the original image encoding feature and the reference image encoding feature, thereby constructing a more semantically relevant second image mapping feature.

[0059] Determining visual context features based on the image features corresponding to the previous feature processing cycle is crucial for maintaining image coherence and consistency during iterative generation. Visual context features capture visual information from the image at the previous moment, providing guidance for generation at the current moment. This can be achieved by performing global average pooling or global max pooling on the image features corresponding to the previous feature processing cycle, compressing them into a fixed-dimensional vector as the visual context features. Alternatively, a small encoder network can be used to process the image features corresponding to the previous feature processing cycle, extracting more representative contextual information.

[0060] When the current feature processing cycle is the first feature processing cycle, the visual context features are determined based on the reference image to provide the model with a clear reference direction at the beginning of the image generation process. Since the first feature processing cycle does not have "image features corresponding to the previous feature processing cycle," directly using the reference image as the visual context ensures that the generation process is guided by the target style or content from the outset. This can be achieved by inputting the reference image into a pre-trained feature extraction network (e.g., VGG, ResNet) to extract its high-level semantic features as visual context features. Alternatively, the encoded features of the reference image can be directly used as the initial visual context.

[0061] The forward velocity field processing module of the image generation model processes the editing information corresponding to the reference image, the visual context features, and the second image mapping features in the image editing information to obtain the initial forward velocity field features, which is a key step in generating the forward velocity field. The forward velocity field processing module is a specially designed neural network module whose function is to predict the changing trend of image features during the "forward" diffusion process based on various input information. This module can be a U-Net architecture or composed of a series of residual blocks. It receives the spliced ​​or fused image editing information, visual context features, and second image mapping features as input, and outputs a tensor representing the velocity field, such as a two-dimensional vector field, through multiple nonlinear transformations, indicating the movement of each pixel or feature point in the x and y directions.

[0062] In its implementation, this application's solution ensures the accuracy and contextual awareness of the forward velocity field features by refining the construction of the second image mapping features and determining the visual context features. First, by integrating the first image mapping features, the original image encoding features, and the reference image encoding features, a second image mapping feature containing multi-source information is constructed, enabling it to comprehensively reflect the relationship between the current image state and the target reference image. Second, by dynamically determining the visual context features based on the image features or reference image corresponding to the previous feature processing cycle, this application's solution provides continuous and accurate visual guidance throughout the iteration process. Especially in the initial stage, it directly utilizes the reference image as context, avoiding the initial blindness. Finally, the forward velocity field processing module of the image generation model uses this rich and accurate input information to generate high-quality initial forward velocity field features, effectively guiding the image features towards the target editing direction. This multi-source information fusion and context-adaptive strategy enables the image generation model to more accurately understand the editing intent and generate a target image highly consistent with the style and content of the reference image.

[0063] For example, suppose we need to edit an original image into an image with a specific style, defined by a reference image. In a given feature processing cycle, the first image mapping feature corresponding to that cycle is first obtained. This feature might be a 64x64x256 feature map, representing the local details of the original image at the current noise level. Simultaneously, the existing original image encoding features and reference image encoding features (e.g., both 128x128x512 feature maps) are also ready. To construct the second image mapping feature, these three feature maps are concatenated along the channel dimension to form a 64x64x(256+512+512) tensor. This tensor is then processed through a residual block containing a 3x3 convolutional layer and a ReLU activation function to obtain a 64x64x512 second image mapping feature. When determining visual context features, if the current period is not the first feature processing cycle, the image features generated in the previous feature processing cycle (e.g., a 128x128x512 tensor) are input into a global average pooling layer to obtain a 512-dimensional vector as the visual context features. If the current period is the first feature processing cycle, the reference image is input into a pre-trained ResNet-50 model, and the feature vector of its penultimate layer is extracted as the visual context features. Finally, the editing information corresponding to the reference image in the image editing information (e.g., a text embedding vector), the visual context features determined above, and the second image mapping features are provided as input to the positive velocity field processing module of the image generation model. This module can be a small U-Net network that fuses these input features and outputs a 64x64x2 tensor, where the two channels represent the velocity components of the image features in the x and y directions, respectively, i.e., the initial positive velocity field features.

[0064] Through the above technical solution, this application can more accurately construct the second image mapping features, integrating the current state of the original image, the global encoding of the original image, and the global encoding information of the reference image, thus providing a more comprehensive input for subsequent velocity field calculations. Simultaneously, by adaptively determining visual context features according to the processing cycle—that is, utilizing the reference image in the first cycle and the image features of the previous cycle in subsequent cycles—the continuity and goal orientation of the image generation process are ensured. This refined feature construction and context-aware mechanism enables the image generation model to more accurately capture editing intentions and generate target images highly consistent with the style and content of the reference image, significantly improving the quality and control of image editing.

[0065] When determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantically aligned velocity field features corresponding to the original image, a challenge may arise in effectively integrating this information to ensure the semantic consistency of the generated image features and to avoid semantic drift or detail distortion during iteration. To address this, in this embodiment, determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantically aligned velocity field features corresponding to the original image includes: The reverse velocity field features and the forward velocity field features are fused to obtain target velocity field features; a preset semantic alignment strategy is used to perform semantic alignment processing on the original image and the image features corresponding to the previous feature processing cycle to obtain semantically aligned velocity field features; and image features corresponding to the current feature processing cycle are generated based on the target velocity field features and the semantically aligned velocity field features.

[0066] Specifically, feature fusion refers to combining two or more feature vectors or feature maps to form a new feature that contains more information or is more representative. Its purpose is to integrate information from different sources or of different natures, thereby providing a more comprehensive input for subsequent processing. For example, the target velocity field feature can be obtained by element-wise addition, element-wise multiplication, channel concatenation, and convolution of the inverse and forward velocity field features, or by dynamically weighting and combining them using a learnable fusion network (such as a multilayer perceptron or a small convolutional neural network). The target velocity field feature refers to the comprehensive velocity information obtained by fusing the inverse and forward velocity field features. This feature characterizes how image features should be moved or transformed holistically from the previous cycle within the current feature processing cycle in response to image editing information. A predefined semantic alignment strategy refers to a series of rules, algorithms, or models defined in advance during image processing to ensure the consistency of semantic content between different images or different features. This strategy aims to identify and match semantic regions or objects in an image, thereby maintaining the integrity and correspondence of these semantic regions during image editing or generation. For example, this strategy may include a deep learning-based semantic segmentation algorithm to extract semantic masks from an image; or it may include a feature matching-based algorithm to identify and align key semantic points or regions in an image. Semantic alignment processing refers to operating on the image features of the original image and the image features corresponding to the previous feature processing cycle according to a preset semantic alignment strategy to calculate the semantic correspondence or differences between them. This processing aims to quantify or describe the adjustments required to ensure that the image features corresponding to the previous feature processing cycle are semantically consistent with the original image. The semantic alignment velocity field feature refers to the velocity information obtained through semantic alignment processing, used to guide the semantic correction of image features. This feature reflects the local or global adjustments that the current image features need to make relative to the original image at the semantic level to ensure the accuracy and consistency of semantic content. Generating the image features corresponding to the current feature processing cycle refers to finally determining the image features that the current feature processing cycle should have after comprehensively considering the target velocity field feature (reflecting the overall transformation) and the semantic alignment velocity field feature (reflecting semantic correction). This step is a crucial step in the iterative process, integrating all relevant information to produce an intermediate image feature that both conforms to the editing intent and maintains semantic consistency.

[0067] Based on this, a semantically aligned velocity field feature is introduced and combined with a target velocity field feature fused from inverse and forward velocity field features to determine the image features corresponding to the current feature processing cycle. Specifically, firstly, the inverse transformation information reflecting the transformation from the original image to the current state and the forward transformation information from the previous cycle to the current state are fused to obtain a preliminary, comprehensive target velocity field feature. This target velocity field feature captures the overall motion trend that the image should follow during the editing process. Simultaneously, to address the semantic drift problem that may occur during iteration, this application utilizes a pre-defined semantic alignment strategy to perform semantic alignment processing on the original image and the image features corresponding to the previous feature processing cycle. This processing aims to identify and quantify the semantic differences between the two, generating a semantically aligned velocity field feature specifically designed to correct and maintain the semantic consistency of the image. Finally, this semantically aligned velocity field feature is combined with the previously obtained target velocity field feature to jointly guide the generation of image features corresponding to the current feature processing cycle. This approach not only considers the dynamic editing needs of images but also ensures that the semantic content of the image remains highly consistent with the original image throughout the entire iterative generation process. This effectively avoids semantic drift and detail distortion, resulting in more accurate and stable image features.

[0068] For example, when determining the image features corresponding to the current feature processing cycle, the inverse velocity field features and the forward velocity field features can first be concatenated and feature extracted through a convolutional layer to obtain the target velocity field features. For instance, the two velocity field features can be stacked along the channel dimension and then input into a feature fusion network containing multiple convolutional layers and activation functions. This network learns how to effectively combine this information. Simultaneously, for semantic alignment processing, a pre-trained semantic segmentation model (e.g., a model based on U-Net or DeepLabV3+ architecture) can be used as the preset semantic alignment strategy. This model can perform semantic segmentation on the original image and the image features corresponding to the previous feature processing cycle, respectively, to obtain their respective semantic masks. Subsequently, an optical flow estimation network (e.g., RAFT or PWC-Net) can be used to calculate the pixel-level correspondence between the semantic mask of the image features corresponding to the previous feature processing cycle and the semantic mask of the original image, thereby obtaining the semantically aligned velocity field features. Finally, when generating the image features corresponding to the current feature processing cycle, the target velocity field features and the semantically aligned velocity field features can be combined by a weighted summation, or the contributions of the two can be dynamically adjusted by a gating mechanism (such as an attention mechanism). Then, the combined velocity field is applied to the image features corresponding to the previous feature processing cycle, for example by a differentiable sampling operation (such as bilinear interpolation), thereby obtaining the image features corresponding to the current feature processing cycle.

[0069] Through the above technical solution, in the iterative generation of image features, not only can inverse and forward velocity field information be effectively integrated to capture the dynamic editing needs of the image, but also, by introducing semantically aligned velocity field features, the semantic consistency between the generated image features and the original image can be significantly enhanced. This effectively solves the problems of semantic drift and detail distortion that may occur in multi-step iterative editing, ensuring that the final generated target image is more semantically accurate, coherent, and has higher visual quality.

[0070] Based on this, considering the complexity of image content or the diversity of editing needs, inaccurate semantic alignment may occur, thus affecting the quality and realism of the final generated image. Therefore, in this embodiment, the semantic alignment process is performed on the original image and the image features corresponding to the previous feature processing cycle using a preset semantic alignment strategy to obtain semantic alignment velocity field features, including: According to a preset semantic alignment strategy, the original image and the image features corresponding to the previous feature processing cycle are semantically segmented to obtain pixel information, and semantic alignment velocity field features are constructed based on the pixel information.

[0071] Specifically, semantic segmentation refers to segmenting pixel information corresponding to semantic information that does not need modification from the original image and the image features corresponding to the previous feature processing cycle. Its purpose is to reduce the instability caused by global modifications in image editing operations. Correspondingly, pixel information refers to the pixel information that does not need modification determined from the original image and the image features corresponding to the previous feature processing cycle after semantic segmentation. It can be the position information of the pixel in the image, or the unique identification information corresponding to each pixel. This embodiment does not make any limitations here.

[0072] In practical implementation, the semantic alignment strategy can encapsulate relevant algorithms for semantic segmentation and semantic alignment velocity field feature construction, enabling the rapid implementation of semantic segmentation and feature construction operations using specific algorithms. Semantic segmentation can be achieved through algorithms such as Concept Attention, Non-local Neural Networks, CCNet (Criss-Cross Attention Network), or DANet (Dual Attention Network), thereby ensuring accurate extraction of pixel information from images and features for the construction of semantic alignment velocity field features.

[0073] Based on this, in order to ensure the quality and realism of the final generated image when constructing semantic alignment velocity field features, the original image and the image features corresponding to the previous feature processing cycle can be semantically segmented according to a preset semantic alignment strategy to obtain pixel information. After obtaining the pixel information that does not need to be modified, the semantic alignment velocity field features can be constructed by combining the pixel information for subsequent image editing operations.

[0074] Based on this, simply adopting a preset semantic alignment strategy may not be sufficient to address the challenges posed by the complexity of image content or the diversity of editing needs, potentially leading to inaccurate semantic alignment and consequently affecting the quality and realism of the final generated image. Therefore, in this embodiment, the step of semantically segmenting the original image and the image features corresponding to the previous feature processing cycle according to the preset semantic alignment strategy to obtain pixel information, and constructing a semantic alignment velocity field feature based on the pixel information, includes: A semantic segmentation algorithm and an image enhancement algorithm are determined according to a preset semantic alignment strategy; the original image and the image features corresponding to the previous feature processing cycle are processed based on the semantic segmentation algorithm to obtain the first pixel information corresponding to the original image and the second pixel information of the image features corresponding to the previous feature processing cycle; the first pixel information and the second pixel information are processed based on the image enhancement algorithm, and the semantic alignment velocity field features are determined according to the processing results.

[0075] Specifically, a pre-defined semantic alignment strategy refers to a set of rules, methods, or algorithm combinations pre-set to guide the semantic alignment process before image processing, based on specific application scenarios and editing goals. This strategy may include weight allocation for different semantic regions, alignment accuracy requirements, and the selection of appropriate semantic analysis and processing tools. For example, this strategy might stipulate that when editing faces, priority should be given to maintaining the semantic consistency of key features such as eyes, nose, and mouth; or when replacing backgrounds, it should ensure that the semantic boundaries between foreground objects and the new background are clear and blend naturally. Semantic segmentation algorithms are a computer vision technique that classifies each pixel in an image into a predefined semantic category, thereby achieving pixel-level understanding of the image content. Through semantic segmentation, different objects, regions, or scenes in an image can be accurately identified, and a semantic label can be assigned to each pixel. For example, semantic segmentation algorithms can identify different semantic regions in an image, such as "sky," "person," "car," and "road." Common semantic segmentation algorithms include Fully Convolutional Networks (FCN), U-Net, and DeepLab series models, which learn local and global features of images through deep learning techniques to achieve high-precision pixel-level classification. Image enhancement algorithms aim to improve the visual quality of images, making them more suitable for subsequent processing or human observation. Their functions typically include increasing image contrast, sharpening edges, removing noise, and adjusting brightness or color balance. In the context of semantic alignment, image enhancement algorithms can be used to highlight the boundaries of semantic regions, smooth textures within regions, or adjust the representation of pixel information for better comparison and alignment. For example, image enhancement algorithms can use histogram equalization to enhance image contrast, or use Gaussian filtering, bilateral filtering, and other methods to smooth image noise while preserving important edge information. The first and second pixel information refer to the pixel-level semantic representations obtained after processing the original image and the image features corresponding to the previous feature processing cycle using a semantic segmentation algorithm, respectively. This information typically exists in the form of semantic masks, pixel category label maps, or feature maps, where each pixel is associated with its semantic category or has a feature value representing its semantic attributes. For example, the first pixel information can be a matrix of the same size as the original image, where each element represents the semantic category of the corresponding pixel in the original image (e.g., "sky" is 1, "person" is 2, etc.); the second pixel information represents the pixel-level semantic content of the image features corresponding to the previous feature processing cycle in a similar way.

[0076] Based on this, the semantic alignment velocity field features are obtained by refining the execution process of the semantic alignment strategy. Specifically, the scheme first determines the semantic segmentation algorithm and image enhancement algorithm to be used according to the preset semantic alignment strategy. This explicit algorithm selection ensures the professionalism and specificity of the semantic alignment process. Subsequently, the determined semantic segmentation algorithm is used to perform pixel-level processing on the original image and the image features corresponding to the previous feature processing cycle, thereby obtaining the first pixel information corresponding to the original image and the second pixel information corresponding to the image features of the previous feature processing cycle, respectively. In this way, the system can deeply understand the semantic composition of the two images at the pixel level, laying the foundation for subsequent accurate alignment. Next, the first and second pixel information are processed based on the determined image enhancement algorithm. The role of image enhancement at this stage is to optimize these pixel-level semantic representations, for example, by sharpening semantic boundaries or adjusting the intensity of semantic features to make them more suitable for comparison and fusion. Finally, the semantic alignment velocity field features are determined based on the enhanced first and second pixel information. This process, which acquires fine-grained pixel information through semantic segmentation and then optimizes this information through image enhancement, enables semantic alignment velocity field features to more accurately reflect the semantic correspondence and differences between the original image and the image features corresponding to the previous feature processing cycle. When this refined semantic alignment velocity field feature is fused with inverse and forward velocity field features, it can significantly improve the semantic consistency and detail preservation ability of the image features corresponding to the current feature processing cycle, thereby achieving more natural semantic transformation and content generation that better aligns with user intent during image editing.

[0077] For example, when determining the preset semantic alignment strategy, this strategy can be set to prioritize the semantic integrity and boundary clarity of the main objects in the image. Based on this strategy, the DeepLabV3+ model can be selected as the semantic segmentation algorithm, which is widely recognized for its high-precision semantic segmentation capabilities in complex scenes. Simultaneously, an image enhancement algorithm combining Adaptive Histogram Equalization (AHE) and bilateral filtering can be adopted. AHE is used to improve local contrast, while bilateral filtering is used to effectively preserve edge details while smoothing regions. In actual processing, the original image and the image features corresponding to the previous feature processing cycle are first input into the DeepLabV3+ model. The model outputs two sets of pixel-level semantic category maps, serving as the first and second pixel information, respectively. For example, if the image contains "person," "sky," and "building," the output pixel information will label the category to which each pixel belongs. Subsequently, these semantic category maps are input into the image enhancement algorithm. For example, edge enhancement processing can be performed on the semantic category maps to make the boundaries between different semantic regions clearer, or weighted processing can be applied to the features of specific semantic regions. Finally, based on the enhanced first and second pixel information, semantically aligned velocity field features are generated by calculating their semantic differences and correspondences. For example, the velocity field can be constructed by calculating the semantic category difference between the two semantic category maps at each pixel location and combining it with the geometric transformation information of the semantic region.

[0078] Through the above technical solution, this application can effectively solve the problem of inaccurate semantic alignment during image editing. By introducing a semantic segmentation algorithm, fine pixel-level semantic analysis can be performed on the original image and the image features corresponding to the previous feature processing cycle, thereby obtaining accurate first and second pixel information, which greatly improves the depth of understanding of the semantic content of the image. Furthermore, by processing these pixel information through image enhancement algorithms, the expression of semantic features can be optimized, such as sharpening semantic boundaries or highlighting key semantic regions, making subsequent semantic alignment more accurate and robust. Therefore, the determined semantic alignment velocity field feature can more accurately reflect the correspondence and change requirements of the image at the semantic level. When this high-precision semantic alignment velocity field feature is combined with the inverse velocity field feature and the forward velocity field feature, it can ensure that when generating the image features corresponding to the current feature processing cycle, not only the overall structure and texture changes of the image are considered, but also the semantic content of the image is more accurately maintained or transformed, avoiding the problems of semantic misalignment or unnatural fusion, thereby significantly improving the semantic consistency, visual realism, and accuracy of the editing effect of the final generated image.

[0079] In other words, see Figure 2aThe schematic diagram illustrates that the image processing method provided in this embodiment, in order to enable multi-image editing operations based on text commands, can employ the processing logic of the aforementioned image generation model to perform image editing operations, thereby obtaining a target image that meets the usage requirements. Specifically, assuming that the initial time when the algorithm starts execution is t=1, and the time when the algorithm completes is t=0, while the time t gradually decreases from 1 to 0 during the editing process. Simultaneously, the encoded features corresponding to the original image at the initial time are denoted as... This corresponds to the features obtained after feeding the input image into the encoder of the RF model. Additionally, the image features at each time point during the editing process are denoted as... Based on this, the image generation model can be understood as achieving multi-image editing by combining the forward and reverse velocity fields. Both the forward and reverse velocity fields are obtained through the RF model. At time t, the reverse velocity field is calculated. At that time, the input to the RF model is the first image mapping feature. Simultaneously input the annotation text of the original image (the editing information corresponding to the original image in the image editing information), where It is standard Gaussian noise. And the calculation of the forward velocity field... At that time, the input to the RF model is the second image mapping feature. Features of visual context images And the annotated text of the edited image (the editing information corresponding to the reference image in the image editing information); based on this, through By fusing the positive and negative velocity fields in this way, the target velocity field can be obtained. Based on velocity field It can gradually transform samples from the input image distribution to the target image distribution without explicitly performing a diffusion inversion process.

[0080] Building upon this, to further enhance the consistency between the generated result and the input image, the original image and the sample can be segmented using the ConceptAttention algorithm. In this context, pixels corresponding to semantic information that do not require modification are identified. Subsequently, the DPS algorithm can be used to calculate the semantic alignment velocity field based on the mean square error between these two pixel groups. Finally, by... and The calculations are performed, and the velocity field is used to update the samples. This allows for processing in one cycle, which is then iterated until t gradually decreases to 0. The image features output at t=0 are then input into the decoder for decoding to obtain the target image for use. For example... Figure 2b The reference image and the original image shown can be processed as described above to obtain a fitting effect image of a user wearing specific clothing.

[0081] In practice, the above processing can be accomplished through the following steps: First, the encoder based on the RF model encodes the input image to obtain... Then, at a series of time steps t, the following steps are executed cyclically: 1. Sample K Gaussian noises Calculate separately . 2. Based on K Gaussian noise corresponding to ,calculate . 3. For group K and The reverse velocity field was calculated using the RF model. and positive velocity field The calculation of the forward velocity field utilizes both the labeled text of the target image and visual context features. Therefore, the editing results can be controlled by text commands and visual context images. 4. Average the K groups of forward and reverse velocity fields respectively, and calculate based on the averaged forward and reverse velocity fields. . 5. Calculate the semantic alignment velocity field based on the ConceptAttention algorithm and the DPS algorithm. . 6. According to Calculate the sample corresponding to the next time step . The loop ends when t=0. The decoder corresponding to the RF model is then used. Decode it into an image, and you will get the edited image.

[0082] The image processing method provided in this embodiment, by combining a semantic segmentation algorithm (ConceptAttention algorithm) and an image enhancement algorithm (DPS algorithm), can demonstrate significant editing advantages in image generation or editing tasks. Specifically, the ConceptAttention algorithm can accurately identify pixels in the original image and samples that correspond to semantic information that does not need modification. This characteristic provides a crucial foundation for subsequent processing, clarifying which regions should maintain semantic consistency and which regions can be adjusted. The DPS algorithm, based on these unmodified pixels, generates a semantic alignment velocity field by calculating the mean square error between the original image and the samples. This velocity field reflects the direction and degree of adjustment required for the samples. Finally, by fusing the semantic preservation tendency implicit in the ConceptAttention algorithm with the velocity field generated by DPS, the target velocity field is obtained. Updating the samples with this target velocity field ensures accurate preservation of semantic information and utilizes DPS to achieve flexible adjustments based on semantic alignment, thus improving both the semantic consistency and visual quality of the generated or edited results.

[0083] Furthermore, a one-off processing approach may result in poor image quality or an inability to fully integrate image editing information, especially in complex editing scenarios. It may be difficult to ensure sufficient evolution and convergence of image features, thus affecting the final target image generation. To address this, in this embodiment, after the step of determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image is executed, the process further includes: If the current feature processing cycle does not meet the image generation conditions, the image features corresponding to the current feature processing cycle are used as the image features corresponding to the previous feature processing cycle. Then, the following steps are performed: constructing the inverse velocity field features corresponding to the current feature processing cycle based on the original image encoding features and the image editing information; and constructing the forward velocity field features corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information, and the image features corresponding to the previous feature processing cycle. This process continues until the time step corresponding to the target feature processing cycle is equal to a preset time step. Then, it is determined that the target feature processing cycle meets the image generation conditions, and the image features corresponding to the target feature processing cycle are decoded using the decoding module of the image generation model to obtain the target image corresponding to the image editing information.

[0084] Specifically, "the current feature processing cycle does not meet the image generation conditions" means that at the end of a certain feature processing cycle, the system evaluates that the currently generated image features have not yet reached the expected quality, convergence, or consistency requirements with the edited information. Image generation conditions may include, but are not limited to: the difference between the current image features and the target image features is less than a preset threshold, the gradient change of the image features is less than a preset threshold, or the image features do not meet preset standards in specific indicators (such as perceptual quality, semantic consistency). When these conditions are not met, it indicates that further iterative optimization is needed. "Using the image features corresponding to the current feature processing cycle as the image features corresponding to the previous feature processing cycle" is a key step in realizing iterative processing. It means using the image features calculated in the current cycle as the starting point or reference for the next iteration, so that subsequent feature processing can be progressively optimized based on the previous processing results. This allows information to be transferred and accumulated between different processing cycles, enabling image features to gradually evolve towards meeting the image generation conditions. The step of "constructing the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and constructing the forward velocity field feature corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information, and the image features corresponding to the previous feature processing cycle" describes the re-execution of the steps to construct the inverse and forward velocity field features during the iteration process. Specifically, when constructing the forward velocity field feature, the updated "image features corresponding to the previous feature processing cycle" is used, ensuring that the velocity field calculation is based on the latest image feature state in each iteration, thereby guiding the image features towards a better direction. This repeated calculation is the core mechanism for achieving gradual feature optimization. The termination condition of the iteration process is defined as "determining that the target feature processing cycle satisfies the image generation condition when the time step corresponding to the target feature processing cycle is equal to the preset time step." The time step is a commonly used parameter in diffusion models or iterative optimization processes, and it is usually associated with the number of iterations or processing stages. When the iteration reaches a preset time step (i.e., a preset number of iterations or processing depth), even if other image generation conditions (such as quality thresholds) are not fully met, the current cycle is forcibly considered to have met the image generation conditions. This provides a mechanism to ensure that the algorithm completes processing within a finite time, avoiding infinite loops, and can obtain relatively optimal results under preset computing resources. The step of "decoding the image features corresponding to the target feature processing cycle using the decoding module of the image generation model to obtain the target image corresponding to the image editing information" is the step after the iteration process ends, converting the image features that have finally converged or reached the preset time step into a perceptible image. The decoding module maps the abstract image features back to pixel space to generate the final target image.This step is performed after the iteration termination condition is met, ensuring that only fully optimized and evolved image features are decoded and output.

[0085] Based on this, an iterative processing mechanism is introduced to finely control the image feature generation process. After each feature processing cycle, the system first evaluates whether the image features corresponding to the current feature processing cycle meet the preset image generation conditions. If the current image features have not yet reached the expected quality or convergence state, decoding will not be performed immediately. Instead, the image features obtained in the current cycle will be used as the "image features corresponding to the previous feature processing cycle" for the next iteration. Subsequently, based on this updated "image features corresponding to the previous feature processing cycle," the system will combine the original image encoding features, the reference image encoding features, and image editing information to reconstruct the inverse velocity field features and forward velocity field features corresponding to the current feature processing cycle. In this way, each iteration can use the optimized image features from the previous cycle as input, making the velocity field calculation more accurate, thereby guiding the image features to further evolve and optimize in subsequent cycles. This iterative process will continue until the preset termination condition is reached, that is, the time step corresponding to the target feature processing cycle is equal to the preset time step. At this point, the system determines that the image generation conditions are met and uses the decoding module of the image generation model to decode the final image features, thereby obtaining a high-quality target image that fully integrates editing information. This iterative optimization mechanism enables image features to converge gradually, effectively solving the problem of insufficient image quality or inadequate fusion of editing information that may result from a single processing step, thus ensuring the accuracy and visual effect of the final generated image.

[0086] For example, suppose the image generation conditions are set as follows: the L2 norm change of the image features is less than 0.001 for three consecutive cycles, or the number of processing cycles reaches 500 steps. During image processing, the original image coding features, reference image coding features, and image editing information are first acquired, and initial feature processing is performed to obtain the image features corresponding to the current feature processing cycle. Subsequently, the system checks whether the image features satisfy the condition that the L2 norm change is less than 0.001. If this condition is not met, for example, if the L2 norm change is 0.01, the current image features are used as the image features corresponding to the previous feature processing cycle in the next cycle. Next, the system recalculates the reverse velocity field features and the forward velocity field features, where the calculation of the forward velocity field features utilizes the updated image features corresponding to the previous feature processing cycle. This process is repeated. For example, in the 100th processing cycle, if the L2 norm change still does not meet the condition, but the preset time step has been reached (e.g., the time step is directly related to the number of processing cycles, reaching 500 steps), the system will forcibly determine that the image generation condition is met. At this point, regardless of whether the L2 norm change of the image features has converged to below 0.001, the image features corresponding to the current processing cycle will be input into the decoding module of the image generation model, ultimately generating the target image. This mechanism ensures that a target image optimized through multiple iterations can be obtained within limited computational resources and time.

[0087] Through the above technical solution, this application introduces an iterative optimization mechanism, enabling multiple rounds of fine-tuning in the image feature generation process. When image features fail to meet the expected generation conditions in the current processing cycle, the system can use the current result as the starting point for the next iteration and recalculate the velocity field features, thereby guiding the image features to gradually evolve towards a better direction. This iterative processing method effectively avoids the problems of poor image quality or insufficient fusion of editing information that may result from a single processing step. Simultaneously, by setting a preset time step as a forced termination condition, it ensures that the algorithm can complete the image generation task within limited computing resources and time, improving the robustness and practicality of the method. Ultimately, this scheme can generate target images that are highly consistent with the image editing information and have better visual effects, significantly improving the quality and efficiency of image processing. Furthermore, if the image generation model is not trained with sufficient and effective data, it may face problems such as insufficient generalization ability, poor image quality, or inability to accurately capture editing intent when handling complex image editing tasks, thereby affecting the final target image generation effect and user experience. Therefore, in this embodiment, the training of the image generation model includes: Obtain an image sample set, wherein the image sample set includes sample images, a set number of context images, sample image editing information, and label images; use the image sample set to train an initial image generation model until an image generation model that meets the training stopping condition is obtained.

[0088] Specifically, training an image generation model refers to optimizing its internal parameters by learning from a large amount of data, enabling it to perform specific image generation or editing tasks. The goal of training is to enable the model to learn effective feature representations and mappings from input data, thus generating high-quality, expected outputs given new inputs. For example, supervised learning can be used, guiding model parameter adjustments by comparing the model's predicted output with the true labels; or unsupervised or semi-supervised learning can be used, pre-training with a large amount of unlabeled data and then fine-tuning with a small amount of labeled data. Obtaining an image sample set refers to collecting and preparing a set of data for training the image generation model. The image sample set is the foundation of model learning, and its quality and diversity directly affect model performance. This sample set typically contains data used as input to the model and corresponding expected output data. For example, it can be selected or constructed from public datasets (such as ImageNet, COCO, FFHQ, etc.) or obtained through data collection for specific scenarios. During the construction process, image preprocessing may be required, such as resizing, normalization, and data augmentation, to adapt to the model's input requirements and improve its generalization ability. The image sample set includes sample images, a predetermined number of context images, sample image editing information, and label images. Sample images refer to the original images used as one of the model's training inputs, upon which the model will edit or generate. For example, this could be a user-provided source image to be edited. The predetermined number of context images refers to images associated with the sample images that provide additional background or semantic information. These images help the model understand the editing context, resulting in more coherent and natural edits. The predetermined number can be adjusted according to the needs of the training task; for example, it could be one or more images similar to or complementary to the content of the sample images. Sample image editing information refers to instructions or data describing what kind of editing operation to perform on the sample image. This information can be text descriptions (e.g., "dye hair red"), semantic masks, sketches, style reference images, etc. It is a key input for the model to understand the editing intent. Label images refer to the target image after ideal editing, corresponding to the sample image editing information. Label images serve as the ground truth during training, used to calculate the model's loss and guide the optimization of model parameters. For example, if the sample image editing information is "dye hair red," then the label image is the sample image with the hair dyed red. Training the initial image generation model using the image sample set refers to inputting the prepared image sample set into an untrained or initially initialized image generation model, and adjusting the model's internal parameters through an iterative optimization process. The purpose of training is to enable the model to learn the mapping relationship from input (sample image, context image, editing information) to output (label image).For example, gradient descent and its variants (such as Adam and SGD) can be used as optimization algorithms to update the model's weights and biases by reducing the difference between the predicted and labeled images (e.g., using L1 loss, L2 loss, perceptual loss, or adversarial loss). The phrase "until a training stopping condition is met" means that the training process is not indefinite but requires setting certain conditions to determine when to stop. The training stopping condition aims to prevent overfitting or underfitting and ensure that the model reaches the expected performance level. For example, the training stopping condition may include: reaching a preset number of training epochs, performance metrics on the validation set (such as accuracy, FID score, PSNR, etc.) no longer improving over several consecutive epochs, the loss function value falling below a preset threshold, or the update magnitude of the model parameters being less than a certain minimum value. Once any of these conditions is met, the training process terminates, and the model with the best current performance is saved as the final image generation model.

[0089] Based on this, a systematic training process is employed to ensure the efficiency and accuracy of the image generation model in subsequent image processing tasks. First, a rich and structured data foundation is provided for the model's learning by acquiring image sample sets containing sample images, a set number of context images, sample image editing information, and label images. This data collectively constitutes the key information for the model to understand image content, editing intent, and expected output. Subsequently, the initial image generation model is iteratively trained using these carefully prepared image sample sets. During training, the model continuously learns how to generate target images highly consistent with the label images based on the input sample images, context images, and editing information. This training mechanism enables the image generation model to capture complex image editing rules and semantic correspondences. By continuously optimizing the model parameters until a preset training stopping condition is met, it is ensured that the model not only accurately performs image editing tasks but also possesses good generalization ability, capable of handling diverse editing needs. This training process provides a high-quality image generation model for the aforementioned image processing method, enabling the method to more accurately understand image content and editing intent in subsequent steps such as encoding, feature construction, feature fusion, and decoding. This results in the generation of target images that better meet user expectations, effectively solving the problems of poor image quality and inaccurate capture of editing intent that may result from insufficiently trained models.

[0090] For example, training an image generation model can proceed as follows: First, acquire a set of image samples, which can contain tens or even hundreds of thousands of data pairs. For instance, for an editing task like "change hair color," the sample image could be a photograph of a person, a set number of context images could be photographs of the same person under different lighting or poses, the sample image editing information could be the text instruction "dye hair blue," and the label image could be a photograph of the person's hair precisely dyed blue. This data can come from public datasets, such as CelebA-HQ, and the editing information and label images can be generated through manual or semi-automatic annotation. During the training phase, an initial image generation model based on a diffusion model can be used, such as a denoising diffusion model based on the U-Net architecture. The training process can be set to 100 epochs, using the Adam optimizer with a learning rate of 0.0001. The loss function can combine L1 loss, perceptual loss, and adversarial loss to ensure pixel-level accuracy, visual quality, and realism of the generated images. Training can be stopped when the FID (Fréchet Inception Distance) score calculated on the validation set does not significantly decrease over 10 consecutive epochs, or when the required number of training epochs is reached. When these conditions are met, training stops, and the best-performing model is saved as the final image generation model.

[0091] Through the above technical solution, this application provides a systematic image generation model training method. This method acquires an image sample group containing sample images, a set number of context images, sample image editing information, and label images. This sample group is then used to train an initial image generation model until a training stopping condition is met. This allows the image generation model to fully learn the complex rules and semantic correspondences of image editing, significantly improving the model's ability to understand editing intentions and generate high-quality target images. Therefore, in subsequent image processing, the trained model can more accurately encode the original and reference images, more effectively construct inverse and forward velocity field features, and ultimately generate target images that are highly consistent with the image editing information and have excellent visual effects. This effectively avoids problems such as low image generation quality and unnatural editing effects caused by insufficient model training.

[0092] Furthermore, if the input information is not properly processed and organized, it may lead to low model training efficiency, poor image quality, and difficulty in accurately reflecting the expected editing effect. Therefore, in this embodiment, training the initial image generation model using the image sample set until a training stopping condition is obtained includes: The sample image and the set number of context images are subjected to diffusion processing to obtain a set number of image noise features; the set number of image noise features are spliced ​​using a preset mask matrix to obtain target splicing features; the target splicing features and the sample image editing information are sampled using the initial image generation model to obtain a predicted edited image; the initial image generation model is optimized based on the predicted edited image and the label image until the image generation model that meets the training stopping condition is obtained.

[0093] Specifically, diffusion processing is a commonly used technique in image generation, especially in diffusion models, where it progressively adds noise to an image or progressively recovers an image from noise. During the training phase, this processing is typically used to progressively transform the original image or its features into noisy versions at different time steps to simulate the reverse diffusion process. This processing can be achieved by progressively adding Gaussian noise to the image or its encoded features, adding noise of different intensities or distributions at each time step; alternatively, it can be achieved by processing the image using a pre-trained diffusion encoder that maps the image to a noise feature representation in the latent space. Image noise features are the image representation obtained after diffusion processing; they contain information from the original image but are perturbed by noise. In diffusion models, these features are typically used to train the model to predict noise or denoising directions. Image noise features can be tensors of the same or different dimensions as the original image, containing pixel values ​​or feature values ​​perturbed by noise; or they can be vector representations in the latent space, mapped to this space by the encoder before noise is added.

[0094] A pre-defined mask matrix is ​​a binary or floating-point matrix used to selectively preserve or mask certain parts of the input features, or to indicate the position and weight of different features during concatenation. When concatenating multiple features, the mask matrix ensures that different features are correctly placed and combined. This mask matrix can be a two-dimensional or three-dimensional binary matrix, where "1" indicates that the feature at the corresponding position is preserved, and "0" indicates masking; it can also be a floating-point matrix whose values ​​represent the weights of the corresponding features, used for weighted concatenation. Concatenation connects multiple independent feature tensors along a certain dimension to form a larger feature tensor, allowing the model to process information from different sources simultaneously. Concatenation can be performed along the channel dimension, such as stacking two feature maps with the same spatial resolution; it can also be performed along the batch or sequence dimension, depending on the structure of the features and the model's input requirements. The target concatenated feature is the comprehensive feature representation obtained after concatenation guided by the mask matrix. It contains noise information from the sample image and the context image and is presented to the model in a structured manner. The target stitching feature can be a multi-channel feature map, where different channels correspond to image noise features from different sources; or it can be a flattened vector, where different parts correspond to the stitching result of different features.

[0095] The initial image generation model is an untrained image generation model that receives input features (such as stitching features and editing information) and attempts to generate an image. During training, the model continuously optimizes its parameters to improve generation quality. This model can be a diffusion model based on the U-Net architecture, whose input is noisy features and conditional information, and whose output is predicted noise or denoised features; or it can be the generator part of a Generative Adversarial Network (GAN), whose input is random noise and conditional information, and whose output is the generated image. Sampling processing refers to generating new samples from the data distribution learned by the generative model. In diffusion models, this typically involves starting with pure noise and gradually generating a clear image by iteratively applying the model's denoising capabilities. Sampling processing can be the inverse sampling process in algorithms such as DDPM (Denoising Diffusion Probabilistic Models) or DDIM (Denoising Diffusion Implicit Models); or it can be the direct forward propagation process of the generator, mapping the input to the output image. The predicted edited image is the image generated by the initial image generation model based on the input (target stitching features and sample image editing information). During the training phase, it will be compared with the labeled image to calculate the loss and guide model optimization. The predicted edited image can be an RGB image with the same resolution and number of channels as the original image; or it can be an image representation in the latent space, which needs to be converted into a pixel-space image by the decoder later.

[0096] Label images are the real target images in the training data that correspond to the editing information of the sample images. They represent the ideal result that the model should generate. Label images can be manually annotated or generated through other image processing methods, and have the same format and content as the predicted edited images; they can also be reference images verified by experts that meet specific editing requirements. Optimization refers to improving model performance by adjusting the model's internal parameters (such as weights and biases) to reduce the difference between the predicted edited images and the label images (i.e., the loss function). Optimization can use gradient descent and its variants (such as Adam and SGD) optimizers to update model parameters based on the gradient of the loss function; it can also employ more complex optimization strategies such as reinforcement learning or adversarial training. Training stopping conditions are preset criteria used to determine whether model training should terminate. This prevents the model from overfitting or continuing to waste computational resources when performance no longer improves. Training stopping conditions can be reaching a preset number of training epochs; the loss function on the validation set not decreasing for several consecutive epochs; or a certain evaluation metric (such as FID or PSNR) reaching a preset threshold.

[0097] Based on this, a predetermined number of image noise features are obtained by performing diffusion processing on the sample images and a set number of context images during the training phase. These noise features are then concatenated using a pre-defined mask matrix to form the target concatenated features. This processing method allows the initial image generation model to receive a structured and information-rich input, which includes noise information from the original image, noise information from the context images, and editing information from the sample images. The model performs sampling processing based on these comprehensive features to generate a predicted edited image. Subsequently, the predicted edited image is compared with the labeled image, the loss is calculated, and the model is optimized until a pre-defined training stopping condition is met. This training process ensures that the model can fully learn how to accurately edit and generate images based on editing and context information, thereby significantly improving the training efficiency of the image generation model and the quality of the final generated image.

[0098] For example, suppose we need to train an image generation model that can perform style transfer on images based on text instructions. First, a diffusion process is performed on a sample image (e.g., a landscape photograph) and a set number of context images (e.g., multiple paintings of different artistic styles). For instance, the Denoising Diffusion Probabilistic Models (DDPM) method can be used to progressively add Gaussian noise to these images at multiple time steps, thus obtaining a series of image noise features with different noise levels. Next, using a pre-defined mask matrix, these image noise features are concatenated with the encoded sample image editing information (e.g., "convert the landscape photograph to Van Gogh's Starry Night style"). This mask matrix ensures that noise features from different sources occupy specific regions or channels in the concatenated target feature, thus providing a clear input structure for the model. Subsequently, the initial image generation model receives this target concatenated feature and the sample image editing information and performs a reverse diffusion sampling process to progressively recover the predicted edited image from the noise. Finally, the generated predicted edited image is compared with the corresponding labeled image (i.e., a landscape photo in the style of Van Gogh's Starry Night, processed manually or professionally), and the loss function (e.g., a combination of L1 loss and perceptual loss) is calculated. The Adam optimizer is used to update the model weights based on the gradient of the loss function, and training continues until the loss on the validation set no longer decreases for several consecutive epochs, or reaches a preset threshold for generated image quality evaluation metrics (e.g., FID score). At this point, training stops, and a satisfactory image generation model is obtained.

[0099] In other words, multi-image editing tasks can also be accomplished through large-scale pre-training. Specifically, millions of sample pairs are first collected for training, where each sample pair includes an input image, several context images, a text editing instruction, and a target image. Then, this data is used to train the RF model. On the other hand, multi-image editing without training can be achieved using diffusion inversion. This method first performs diffusion inversion on the input and context images to obtain the corresponding noise vectors. Then, the method uses a mask matrix to combine the noise vectors of all images. Finally, based on the annotations of the target image, this type of method starts from the combined noise vector and performs the sampling process of the RF model again to obtain the edited image.

[0100] Through the above technical solutions, the image generation model can more effectively integrate and utilize multi-source information during training, including the original image, contextual information, and editing instructions. This structured feature processing and training optimization mechanism enables the model to learn more refined and accurate image editing capabilities, thereby generating high-quality, semantically consistent target images that meet user intent in practical applications, significantly improving the flexibility and effectiveness of image editing. The following is in conjunction with the appendix Figure 3 Taking the application of the image processing method provided in this specification in a poster generation scenario as an example, the image processing method will be further explained. Among other things, Figure 3 An interactive schematic diagram of an image processing method provided in one embodiment of this specification is shown, specifically including the following contents.

[0101] The client receives original images, reference images, and poster generation requirements uploaded by users through the image editing interface.

[0102] The client sends the original image, reference image, and poster generation requirements to the server.

[0103] On the server side, the original image, reference image, and poster generation requirements are input into the RF model.

[0104] On the server side, the target poster image is generated based on the RF model processing results, and then the target poster image is sent to the client.

[0105] The client displays the target poster image through the display area of ​​the interactive page.

[0106] The RF model deployed on the server handles image editing operations as follows: First, the encoder based on the RF model encodes the original image to obtain... Then, at a series of time steps t, the following steps are executed cyclically: Sample K Gaussian noises Calculate separately .

[0107] Based on K Gaussian noise corresponding to ,calculate . For group K and The reverse velocity field was calculated using the RF model. and positive velocity field The calculation of the forward velocity field utilizes both the annotation text of the reference image and visual context features. Therefore, the editing results can be controlled by text commands and visual context images. The K groups of forward and reverse velocity fields are averaged separately, and the results are calculated based on the averaged forward and reverse velocity fields. . The semantic alignment velocity field is calculated based on the ConceptAttention algorithm and the DPS algorithm. .

[0108] according to Calculate the sample corresponding to the next time step . If t ≠ 0, iterate the above processing operations until t = 0, at which point the loop ends. Use the decoder corresponding to the RF model to... Decode it into an image to obtain the target poster image.

[0109] In other words, assuming the algorithm starts execution at time t=1 and finishes at time t=0, the time t during the editing process gradually decreases from 1 to 0. Simultaneously, the encoded features corresponding to the original image at the initial time are denoted as... This corresponds to the features obtained after feeding the input image into the encoder of the RF model. Additionally, the image features at each time point during the editing process are denoted as... Based on this, the image generation model can be understood as achieving multi-image editing by combining the forward and reverse velocity fields. Both the forward and reverse velocity fields are obtained through the RF model. At time t, the reverse velocity field is calculated. At that time, the input of the RF model is Simultaneously input the annotation text of the original image, where It is standard Gaussian noise. And the calculation of the forward velocity field... At that time, the input of the RF model is Features of visual context images and the annotated text of the edited image; based on this, through By fusing the positive and negative velocity fields in this way, the target velocity field can be obtained. Based on velocity field It can gradually transform samples from the input image distribution to the target image distribution without explicitly performing a diffusion inversion process.

[0110] To further enhance the consistency between the generated result and the input image, the original image and the sample can be segmented based on the ConceptAttention algorithm. In this context, pixels corresponding to semantic information that do not require modification are identified. Subsequently, the DPS algorithm can be used to calculate the semantic alignment velocity field based on the mean square error between these two pixel groups. Finally, by... and The calculations are performed, and the velocity field is used to update the samples. This allows for processing in one cycle, which is then iterated until t gradually decreases to 0. The image features output at t=0 are then input into the decoder for decoding to obtain the target poster image for user use.

[0111] The image processing method provided in this embodiment, in order to improve the image editing effect, can acquire image editing information, original image, and reference image, and use the encoding module of the image generation model to encode the original image and the reference image respectively to obtain the original image encoding features and the reference image encoding features. Then, based on the original image encoding features and image editing information, the inverse velocity field features corresponding to the current feature processing cycle can be constructed, and based on the reference image encoding features, image editing information, and the image features corresponding to the previous feature processing cycle, the forward velocity field features corresponding to the current feature processing cycle can be constructed. Thus, based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image, the image features corresponding to the current feature processing cycle can be determined. This is then used to process the next feature processing cycle until the target feature processing cycle meets the image generation conditions. At this point, the decoding module of the image generation model can be used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information. This method achieves precise feature fusion and image generation by acquiring image editing information, original image, and reference image, encoding features, constructing inverse and forward velocity field features, combining semantic alignment to determine image features, and decoding to generate target images. It effectively solves the problem of feature alignment in multi-image editing, significantly improves the accuracy and robustness of image editing, and reduces detail distortion.

[0112] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 4 A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification is shown. Figure 4 As shown, the device includes: The encoding module 402 is configured to acquire image editing information, the original image and the reference image, and use the encoding module of the image generation model to encode the original image and the reference image respectively to obtain the original image encoding features and the reference image encoding features; The construction module 404 is configured to construct the inverse velocity field feature corresponding to the current feature processing cycle based on the original image coding features and the image editing information, and to construct the forward velocity field feature corresponding to the current feature processing cycle based on the reference image coding features, the image editing information and the image features corresponding to the previous feature processing cycle; The determination module 406 is configured to determine the image features corresponding to the current feature processing cycle based on the reverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image. The decoding module 408 is configured to decode the image features corresponding to the target feature processing cycle using the decoding module of the image generation model until the target feature processing cycle meets the image generation conditions, thereby obtaining the target image corresponding to the image editing information.

[0113] In an optional embodiment, constructing the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information includes: Based on the original image encoding features, construct multiple first image mapping features corresponding to the current feature processing cycle; based on the multiple first image mapping features and the editing information corresponding to the original image in the image editing information, construct multiple initial reverse velocity field features; determine the reverse velocity field features corresponding to the current feature processing cycle based on the multiple initial reverse velocity field features.

[0114] In an optional embodiment, the construction of any one of the plurality of first image mapping features includes: Determine the Gaussian noise parameters corresponding to the current feature processing cycle, and construct the first image mapping feature corresponding to the current feature processing cycle based on the Gaussian noise parameters and the original image encoding features; wherein, the construction of any one of the plurality of initial inverse velocity field features includes: processing the first image mapping feature and the editing information corresponding to the original image in the image editing information using the inverse velocity field processing module of the image generation model to obtain the initial inverse velocity field feature.

[0115] In an optional embodiment, constructing the positive velocity field feature corresponding to the current feature processing cycle based on the reference image coding features, the image editing information, and the image features corresponding to the previous feature processing cycle includes: Based on the plurality of first image mapping features and the reference image encoding features, a plurality of second image mapping features corresponding to the current feature processing cycle are constructed; based on the plurality of second image mapping features, the image features corresponding to the previous feature processing cycle, and the editing information corresponding to the reference image in the image editing information, a plurality of initial positive velocity field features are constructed; based on the plurality of initial positive velocity field features, the positive velocity field features corresponding to the current feature processing cycle are determined.

[0116] In an optional embodiment, the construction of any one of the plurality of second image mapping features includes: A first image mapping feature corresponding to the current feature processing cycle is determined, and a second image mapping feature corresponding to the current feature processing cycle is constructed based on the first image mapping feature, the original image encoding feature, and the reference image encoding feature. The construction of any one of the plurality of initial positive velocity field features includes: determining a visual context feature based on the image feature corresponding to the previous feature processing cycle, wherein, when the current feature processing cycle is the first feature processing cycle, the visual context feature is determined based on the reference image; and using the positive velocity field processing module of the image generation model, processing the editing information corresponding to the reference image in the image editing information, the visual context feature, and the second image mapping feature to obtain the initial positive velocity field feature.

[0117] In an optional embodiment, determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image includes: The reverse velocity field features and the forward velocity field features are fused to obtain target velocity field features; a preset semantic alignment strategy is used to perform semantic alignment processing on the original image and the image features corresponding to the previous feature processing cycle to obtain semantically aligned velocity field features; and image features corresponding to the current feature processing cycle are generated based on the target velocity field features and the semantically aligned velocity field features.

[0118] In an optional embodiment, the step of performing semantic alignment processing on the original image and the image features corresponding to the previous feature processing cycle using a preset semantic alignment strategy to obtain semantic alignment velocity field features includes: According to a preset semantic alignment strategy, the original image and the image features corresponding to the previous feature processing cycle are semantically segmented to obtain pixel information, and semantic alignment velocity field features are constructed based on the pixel information.

[0119] In an optional embodiment, the step of semantically segmenting the original image and the image features corresponding to the previous feature processing cycle according to a preset semantic alignment strategy to obtain pixel information, and constructing semantic alignment velocity field features based on the pixel information, includes: A semantic segmentation algorithm and an image enhancement algorithm are determined according to a preset semantic alignment strategy; the original image and the image features corresponding to the previous feature processing cycle are processed based on the semantic segmentation algorithm to obtain the first pixel information corresponding to the original image and the second pixel information of the image features corresponding to the previous feature processing cycle; the first pixel information and the second pixel information are processed based on the image enhancement algorithm, and the semantic alignment velocity field features are determined according to the processing results.

[0120] In an optional embodiment, after the step of determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image is executed, the method further includes: If the current feature processing cycle does not meet the image generation conditions, the image features corresponding to the current feature processing cycle are used as the image features corresponding to the previous feature processing cycle. Then, the following steps are performed: constructing the inverse velocity field features corresponding to the current feature processing cycle based on the original image encoding features and the image editing information; and constructing the forward velocity field features corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information, and the image features corresponding to the previous feature processing cycle. This process continues until the time step corresponding to the target feature processing cycle is equal to a preset time step. Then, it is determined that the target feature processing cycle meets the image generation conditions, and the image features corresponding to the target feature processing cycle are decoded using the decoding module of the image generation model to obtain the target image corresponding to the image editing information.

[0121] In an optional embodiment, training the image generation model includes: Obtain an image sample set, wherein the image sample set includes sample images, a set number of context images, sample image editing information, and label images; use the image sample set to train an initial image generation model until an image generation model that meets the training stopping condition is obtained.

[0122] In an optional embodiment, training the initial image generation model using the image sample set until an image generation model that meets the training stopping condition is obtained includes: A diffusion process is performed on the sample image and the set number of context images to obtain a set number of image noise features; the set number of image noise features are then stitched together using a preset mask matrix to obtain the target stitched features; The initial image generation model is used to sample and process the target stitching features and the sample image editing information to obtain a predicted edited image; the initial image generation model is optimized based on the predicted edited image and the label image until the image generation model that meets the training stopping condition is obtained.

[0123] The image processing apparatus provided in this embodiment, in order to improve the image editing effect, can acquire image editing information, an original image, and a reference image, and use the encoding module of the image generation model to encode the original image and the reference image respectively to obtain the original image encoding features and the reference image encoding features. Then, based on the original image encoding features and the image editing information, the inverse velocity field features corresponding to the current feature processing cycle can be constructed, and based on the reference image encoding features, the image editing information, and the image features corresponding to the previous feature processing cycle, the forward velocity field features corresponding to the current feature processing cycle can be constructed. Thus, based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image, the image features corresponding to the current feature processing cycle can be determined. This is then used to process the next feature processing cycle until the target feature processing cycle meets the image generation conditions. At this point, the decoding module of the image generation model can be used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information. This method achieves precise feature fusion and image generation by acquiring image editing information, original image, and reference image, encoding features, constructing inverse and forward velocity field features, combining semantic alignment to determine image features, and decoding to generate target images. It effectively solves the problem of feature alignment in multi-image editing, significantly improves the accuracy and robustness of image editing, and reduces detail distortion.

[0124] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0125] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0126] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0127] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0128] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0129] The processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0130] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image processing method described above.

[0131] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0132] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image processing method described above.

[0133] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0134] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image processing method described above.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0137] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Image editing information, original image, and reference image are acquired, and the encoding module of the image generation model is used to encode the original image and the reference image respectively to obtain the encoding features of the original image and the encoding features of the reference image. Construct the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and construct the forward velocity field feature corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information and the image features corresponding to the previous feature processing cycle; Based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image, the image features corresponding to the current feature processing cycle are determined; Until the target feature processing cycle meets the image generation conditions, the decoding module of the image generation model is used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information.

2. The image processing method according to claim 1, wherein constructing the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information comprises: Based on the original image encoding features, construct multiple first image mapping features corresponding to the current feature processing cycle; Based on the multiple first image mapping features and the editing information corresponding to the original image in the image editing information, multiple initial reverse velocity field features are constructed; The reverse velocity field feature corresponding to the current feature processing cycle is determined based on the multiple initial reverse velocity field features.

3. The image processing method according to claim 2, wherein the construction of any one of the plurality of first image mapping features comprises: Determine the Gaussian noise parameters corresponding to the current feature processing cycle, and construct the first image mapping features corresponding to the current feature processing cycle based on the Gaussian noise parameters and the original image coding features; The construction of any one of the plurality of initial reverse velocity field features includes: The image generation model's inverse velocity field processing module is used to process the first image mapping features and the editing information corresponding to the original image in the image editing information to obtain the initial inverse velocity field features.

4. The image processing method according to claim 2 or 3, wherein constructing the positive velocity field feature corresponding to the current feature processing cycle based on the reference image coding features, the image editing information, and the image features corresponding to the previous feature processing cycle includes: Based on the plurality of first image mapping features and the reference image coding features, a plurality of second image mapping features corresponding to the current feature processing cycle are constructed; Based on the multiple second image mapping features, the image features corresponding to the previous feature processing cycle, and the editing information corresponding to the reference image in the image editing information, multiple initial positive velocity field features are constructed. The positive velocity field feature corresponding to the current feature processing cycle is determined based on the multiple initial positive velocity field features.

5. The image processing method according to claim 4, wherein the construction of any one of the plurality of second image mapping features comprises: Determine the first image mapping feature corresponding to the current feature processing cycle, and construct the second image mapping feature corresponding to the current feature processing cycle based on the first image mapping feature, the original image encoding feature, and the reference image encoding feature; The construction of any one of the plurality of initial positive velocity field features includes: Visual context features are determined based on the image features corresponding to the previous feature processing cycle, wherein, when the current feature processing cycle is the first feature processing cycle, the visual context features are determined based on the reference image; Using the forward velocity field processing module of the image generation model, the editing information corresponding to the reference image, the visual context features, and the second image mapping features in the image editing information are processed to obtain the initial forward velocity field features.

6. The image processing method according to claim 1, wherein determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image includes: The reverse velocity field features and the forward velocity field features are fused to obtain the target velocity field features; Using a preset semantic alignment strategy, semantic alignment processing is performed on the original image and the image features corresponding to the previous feature processing cycle to obtain semantic alignment velocity field features; Based on the target velocity field features and the semantically aligned velocity field features, the image features corresponding to the current feature processing cycle are generated.

7. The image processing method according to claim 6, wherein the step of performing semantic alignment processing on the original image and the image features corresponding to the previous feature processing cycle using a preset semantic alignment strategy to obtain semantic alignment velocity field features includes: According to a preset semantic alignment strategy, the original image and the image features corresponding to the previous feature processing cycle are semantically segmented to obtain pixel information, and semantic alignment velocity field features are constructed based on the pixel information.

8. The image processing method according to claim 6, wherein the step of semantically segmenting the original image and the image features corresponding to the previous feature processing cycle according to a preset semantic alignment strategy to obtain pixel information, and constructing a semantic alignment velocity field feature based on the pixel information, comprises: The semantic segmentation algorithm and image enhancement algorithm are determined based on the preset semantic alignment strategy; Based on the semantic segmentation algorithm, the original image and the image features corresponding to the previous feature processing cycle are processed to obtain the first pixel information corresponding to the original image and the second pixel information of the image features corresponding to the previous feature processing cycle. The image enhancement algorithm is used to process the first pixel information and the second pixel information, and the semantic alignment velocity field features are determined based on the processing results.

9. The image processing method according to claim 1, after the step of determining the image features corresponding to the current feature processing cycle based on the inverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image is executed, it further includes: If the current feature processing cycle does not meet the image generation conditions, the image features corresponding to the current feature processing cycle are used as the image features corresponding to the previous feature processing cycle. Then, the steps of constructing the reverse velocity field features corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and constructing the forward velocity field features corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information, and the image features corresponding to the previous feature processing cycle are executed. Until the time step corresponding to the target feature processing cycle is equal to the preset time step, it is determined that the target feature processing cycle meets the image generation condition, and the decoding module of the image generation model is used to decode the image features corresponding to the target feature processing cycle to obtain the target image corresponding to the image editing information.

10. The image processing method according to claim 1, wherein training the image generation model comprises: Obtain an image sample group, wherein the image sample group includes sample images, a set number of context images, sample image editing information, and label images; The initial image generation model is trained using the image sample set until the image generation model that meets the training stopping condition is obtained.

11. The image processing method according to claim 10, wherein training the initial image generation model using the image sample set until obtaining the image generation model that satisfies the training stopping condition comprises: Diffusion processing is performed on the sample image and the predetermined number of context images to obtain a predetermined number of image noise features; The target splicing features are obtained by splicing the predetermined number of image noise features using a preset mask matrix. The initial image generation model is used to sample and process the target stitching features and the sample image editing information to obtain a predicted edited image; The initial image generation model is optimized based on the predicted edited image and the labeled image until the image generation model that meets the training stopping condition is obtained.

12. An image processing apparatus, comprising: The encoding module is configured to acquire image editing information, the original image, and the reference image, and to encode the original image and the reference image using the encoding module of the image generation model, respectively, to obtain the original image encoding features and the reference image encoding features. The construction module is configured to construct the inverse velocity field feature corresponding to the current feature processing cycle based on the original image encoding features and the image editing information, and to construct the forward velocity field feature corresponding to the current feature processing cycle based on the reference image encoding features, the image editing information and the image features corresponding to the previous feature processing cycle; The determination module is configured to determine the image features corresponding to the current feature processing cycle based on the reverse velocity field features, the forward velocity field features, and the semantic alignment velocity field features corresponding to the original image. The decoding module is configured to decode the image features corresponding to the target feature processing cycle using the decoding module of the image generation model until the target feature processing cycle meets the image generation conditions, thereby obtaining the target image corresponding to the image editing information.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images