Image processing method

Abstract

This specification provides an image processing method, which includes: determining visual features and pixel features corresponding to an original sample image, and processing the original sample image using a visual encoder to be trained to obtain basic features; constructing visual semantic features based on the visual features and the basic features, and constructing pixel semantic features based on the pixel features and the basic features; fusing the visual semantic features and the pixel semantic features, and decoding the fusion result using a pixel decoder to obtain a reconstructed image corresponding to the original sample image; and training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain a target visual encoder.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to image processing methods. Background Technology

[0002] With the development of computer and internet technologies, the field of computer vision has ushered in a revolutionary wave driven by multimodal large language models. These models have shown breakthrough progress in image understanding tasks, capable of fusing visual and linguistic information to achieve more accurate semantic parsing. However, the industry still faces significant technical bottlenecks in building a unified model that can both generate high-quality images and possess accurate understanding capabilities. The three current mainstream technical approaches all have significant drawbacks: The first approach uses full parameter fine-tuning of the pre-trained visual encoder to adapt to the generation task. While this significantly improves the visual quality of the reconstructed image, the fine-tuning process irreversibly destroys the encoder's original general visual semantic representation capabilities. The second approach adopts a strategy of completely freezing the pre-trained encoder, directly utilizing its output high-level semantic features for image reconstruction. Although it retains the understanding capability, because the visual encoder is designed to extract global semantic information, its feature compression process systematically loses pixel-level details, such as texture structure and edge sharpness. The third approach introduces an external strong visual representation model as a guiding module for the generation process. While this improves generation quality by supplementing detailed information, it introduces three major drawbacks: first, it increases the system's dependence on the external model; second, it introduces additional computational overhead and inference latency; and third, the feature representation of the external model struggles to dynamically adapt to changes in the distribution of training data, leading to a significant decrease in system robustness. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of this, embodiments of this specification provide an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a first image processing method is provided, comprising: The visual features and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain basic features; Visual semantic features are constructed based on the visual features and the basic features, and pixel semantic features are constructed based on the pixel features and the basic features; The visual semantic features and the pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image; The target visual encoder is obtained by training the visual encoder to be trained based on the original sample image and the reconstructed image.

[0005] According to a second aspect of the embodiments of this specification, a second image processing method is provided, comprising: The visual features and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain basic features; Visual semantic features are constructed based on the visual features and the basic features, and pixel semantic features are constructed based on the pixel features and the basic features; The visual semantic features and the pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image; The visual encoder to be trained is trained based on the original sample image and the reconstructed image, and the diffusion generation model is optimized based on the trained visual encoder to obtain the target diffusion generation model.

[0006] According to a third aspect of the embodiments of this specification, a third image processing method is provided, comprising: Acquire the image to be processed and identify the image region to be edited in the image to be processed; The target image is obtained by using a target diffusion generation model to edit the image region to be edited in the image to be processed. The target diffusion generation model is determined by the above method.

[0007] According to a fourth aspect of the embodiments of this specification, a fourth image processing method is provided, including: Receive the image to be edited and determine the editing information corresponding to the image to be edited; The target diffusion generation model is used to edit the image to be edited according to the editing information to obtain the target image corresponding to the image to be edited, wherein the target diffusion generation model is determined by the above method.

[0008] According to a fifth aspect of the embodiments of this specification, a fifth image processing method is provided, comprising: Receive the image to be enhanced and determine the sample generation information corresponding to the image to be enhanced; The target diffusion generation model is used to edit the image to be enhanced according to the sample generation information to obtain the sample image corresponding to the image to be enhanced. The target diffusion generation model is determined by the above method.

[0009] According to a sixth aspect of the embodiments of this specification, a first image processing apparatus is provided, comprising: The determination module is configured to determine the visual features and pixel features corresponding to the original sample image, and process the original sample image using the visual encoder to be trained to obtain basic features; The construction module is configured to construct visual semantic features based on the visual features and the basic features, and to construct pixel semantic features based on the pixel features and the basic features; The decoding module is configured to fuse the visual semantic features and the pixel semantic features, and use a pixel decoder to decode the fusion result to obtain the reconstructed image corresponding to the original sample image; The training module is configured to train the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder.

[0010] According to a seventh aspect of the embodiments of this specification, a second image processing apparatus is provided, comprising: The feature determination module is configured to determine the visual features and pixel features corresponding to the original sample image, and process the original sample image using the visual encoder to be trained to obtain basic features; A feature construction module is configured to construct visual semantic features based on the visual features and the basic features, and to construct pixel semantic features based on the pixel features and the basic features; The image decoding module is configured to fuse the visual semantic features and the pixel semantic features, and use a pixel decoder to decode the fusion result to obtain the reconstructed image corresponding to the original sample image; The optimization model module is configured to train the visual encoder to be trained based on the original sample image and the reconstructed image, and optimize the diffusion generation model based on the trained visual encoder to obtain the target diffusion generation model.

[0011] According to an eighth aspect of the embodiments of this specification, a third image processing apparatus is provided, comprising: The image acquisition module is configured to acquire an image to be processed and identify the image region to be edited in the image to be processed; The editing module is configured to use a target diffusion generation model to edit the image region to be edited in the image to be processed, thereby obtaining a target image corresponding to the image to be processed, wherein the target diffusion generation model is determined by the method described above.

[0012] According to a ninth aspect of the embodiments of this specification, a fourth image processing apparatus is provided, comprising: The image receiving module is configured to receive an image to be edited and determine the editing information corresponding to the image to be edited; The editing module is configured to use a target diffusion generation model to edit the image to be edited according to the editing information to obtain a target image corresponding to the image to be edited, wherein the target diffusion generation model is determined by the above method.

[0013] According to a tenth aspect of the embodiments of this specification, a fifth image processing apparatus is provided, comprising: The image receiving module is configured to receive the image to be enhanced and determine the sample generation information corresponding to the image to be enhanced; The editing module is configured to use a target diffusion generation model to edit the image to be enhanced according to the sample generation information to obtain a sample image corresponding to the image to be enhanced, wherein the target diffusion generation model is determined by the above method.

[0014] According to an eleventh aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0015] According to a twelfth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the image processing method described above.

[0016] According to a thirteenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the image processing method described above.

[0017] The image processing method provided in this embodiment, in order to avoid knowledge coverage and detail loss, first determines the visual features and pixel features corresponding to the original sample image during the visual encoder training phase, and then processes the original sample image using the visual encoder to be trained to obtain basic features. Subsequently, visual semantic features are constructed based on the visual features and basic features, and pixel semantic features are constructed based on the pixel features and basic features. Then, the visual semantic features and pixel semantic features are fused, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image. Finally, the visual encoder to be trained is trained based on the original sample image and the reconstructed image to obtain the target visual encoder. This method achieves both generation and understanding tasks by constructing and fusing visual semantic features and pixel semantic features, thus training the visual encoder. It has the advantages of simultaneously achieving high-quality image reconstruction and accurate visual understanding, avoiding knowledge coverage and detail loss, and improving model training efficiency and robustness. Attached Figure Description

[0018] Figure 1 This is a flowchart of a first image processing method provided in one embodiment of this specification; Figure 2a This is a schematic diagram of visual encoder training in an image processing method provided in one embodiment of this specification; Figure 2b This is a schematic diagram of the training of a diffusion generation model in an image processing method provided in one embodiment of this specification; Figure 3 This is a flowchart of a second image processing method provided in one embodiment of this specification; Figure 4 This is a flowchart of a third image processing method provided in one embodiment of this specification; Figure 5 This is a flowchart of a fourth image processing method provided in one embodiment of this specification; Figure 6 This is a flowchart of the fifth image processing method provided in one embodiment of this specification; Figure 7 This is a timing flowchart of a first image processing method provided in one embodiment of this specification; Figure 8 This is a timing flowchart of a second image processing method provided in one embodiment of this specification; Figure 9 This is a timing flowchart of a third image processing method provided in one embodiment of this specification; Figure 10 This is a schematic diagram of the structure of a first image processing apparatus provided in one embodiment of this specification; Figure 11This is a schematic diagram of the structure of a second image processing apparatus provided in one embodiment of this specification; Figure 12 This is a schematic diagram of the structure of a third image processing apparatus provided in one embodiment of this specification; Figure 13 This is a schematic diagram of the structure of a fourth image processing device provided in one embodiment of this specification; Figure 14 This is a schematic diagram of the structure of a fifth image processing apparatus provided in one embodiment of this specification; Figure 15 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0019] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0020] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0022] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0023] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0024] Content Application Platform: This is an application platform that integrates various types of content. The content application platform provides users with a rich variety of multimedia content, including but not limited to live streaming, on-demand video, audio, text and image information, social interaction, shopping, etc.

[0025] Published content: This refers to the content service provided by the content application platform. Published content is any content that users pre-publish on the content platform. For example, the published content can be a recommended note published by the user on the content application platform. The recommended note can include any product, process, tutorial, or related information such as images, videos, descriptions, advantages and disadvantages.

[0026] Large Language Model (LLM): A deep learning-based artificial intelligence model with massive numbers of parameters (usually reaching billions or even hundreds of billions) specifically designed to understand and generate human language.

[0027] DNN (Deep Neural Network) is a machine learning model composed of multiple layers of nonlinear transformation units (neurons), which achieves a complex mapping from input to output by stacking multiple hidden layers.

[0028] Feature distillation is a method that transfers feature knowledge from a teacher model to the training of a student model.

[0029] This specification provides an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0030] In practical applications, the design of visual encoders in traditional image processing systems faces inherent conflicts in feature space. Specifically, when a visual encoder is completely frozen, its output features mainly contain high-level semantic information, lacking low-level pixel-level details, leading to artifacts and blurring during image reconstruction. Conversely, when the visual encoder undergoes full parameter fine-tuning to adapt to the generation task, the general visual semantic representation originally learned by the encoder is overwritten, resulting in performance degradation in understanding tasks such as image classification and object detection. Image understanding tasks require discriminative and sparsity features, while image generation tasks require a continuous, smooth, and easily sampled feature space. This difference in objectives makes it difficult for a single encoder to simultaneously meet both functional requirements, thus affecting the system's key performance indicators in terms of reconstruction quality and understanding ability.

[0031] For example, in the lesion detection stage of an intelligent medical image analysis system, high-precision image reconstruction and disease classification need to be achieved simultaneously. When a frozen visual encoder is used, the soft tissue boundaries in the reconstructed CT images appear blurred, and fine structures such as microcalcifications are lost, affecting doctors' identification of early lesions. If the visual encoder is fine-tuned to optimize reconstruction quality, the discriminative power of feature representation is weakened in subsequent tumor benign / malignant classification tasks, leading to classification biases that may affect the reliability of clinical decisions. Furthermore, in this scenario, the lack of detail in the reconstructed image is directly manifested as insufficient edge sharpness and attenuation of texture information, while the degradation of understanding ability is reflected in the shift of feature space distribution, making it impossible for the system to effectively distinguish similar lesion types. If these problems are not addressed, the image processing system will be unable to achieve a balance between reconstruction quality and understanding ability. In critical application areas, artifacts and blurring in reconstructed images may lead to information omissions, while degradation of understanding ability may cause misjudgments, thereby reducing the overall reliability and applicability of the system. As a result, the system faces functional limitations in actual deployment, making it difficult to meet the dual requirements of multimodal tasks for feature representation, which will restrict the expansion of the technology in scenarios such as medical diagnosis and security monitoring in the long term.

[0032] In view of this, the image processing method provided in this embodiment, in order to avoid knowledge coverage and detail loss, firstly determines the visual features and pixel features corresponding to the original sample image during the visual encoder training stage, and then processes the original sample image using the visual encoder to be trained to obtain basic features; subsequently, visual semantic features are constructed based on the visual features and basic features, and pixel semantic features are constructed based on the pixel features and basic features; then, the visual semantic features and pixel semantic features are fused, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image; finally, the visual encoder to be trained is trained based on the original sample image and the reconstructed image to obtain the target visual encoder. This method achieves the advantages of simultaneously realizing high-quality image reconstruction and accurate visual understanding by constructing and fusing visual semantic features and pixel semantic features, thereby avoiding knowledge coverage and detail loss, and improving model training efficiency and robustness.

[0033] See Figure 1 , Figure 1 A flowchart of a first image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0034] Step S102: Determine the visual features and pixel features corresponding to the original sample image, and process the original sample image using the visual encoder to be trained to obtain basic features.

[0035] Step S104: Construct visual semantic features based on the visual features and the basic features, and construct pixel semantic features based on the pixel features and the basic features.

[0036] Step S106: The visual semantic features and the pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image.

[0037] Step S108: Train the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder.

[0038] The image processing method provided in this embodiment can be applied to the training of a visual encoder in any scenario, enabling the visual encoder to efficiently fuse semantic features and pixel features. The trained visual encoder can be applied to any scenario with image processing requirements, such as image recognition, image editing, and image enhancement. This embodiment does not impose any limitations.

[0039] Specifically, the original sample image refers to the input image used to train the visual encoder. It serves as a true reference for the reconstruction task. Visual features refer to high-level semantic information extracted from the image, typically representing global concepts or objects in the image. This feature focuses on what is contained in the image. Pixel features refer to low-level detail information extracted from the image, such as edges, textures, and fine-grained patterns. This feature focuses on how image details are structured. The visual encoder to be trained refers to a neural network component designed to encode visual information in an image into a compact feature representation. Its parameters can be adjusted during training.

[0040] Correspondingly, basic features refer to the intermediate feature representations obtained after processing the original sample images by the visual encoder to be trained. These features serve as the common foundation for the construction of subsequent semantic features. Visual semantic features refer to feature representations that combine high-level visual semantics with basic features, aiming to capture comprehensive semantic information relevant to the understanding and generation tasks. Pixel semantic features refer to feature representations that combine low-level pixel details with basic features, aiming to preserve fine-grained information crucial for high-quality image reconstruction.

[0041] Correspondingly, a pixel decoder refers to a neural network component responsible for converting a given feature representation back into an image format, typically used to reconstruct images from encoded features. The reconstructed image refers to the output image generated by the pixel decoder based on fused semantic features. This image is designed to faithfully reproduce the original sample image. The target visual encoder refers to the final visual encoder obtained after a training process, optimized for efficient performance in both image understanding and generation tasks.

[0042] Based on this, the visual and pixel features corresponding to the original sample images are determined, and the original sample images are processed using a visual encoder to be trained to obtain basic features. In the image processing, the original sample images are used as input. To comprehensively capture image information, different feature extraction mechanisms can be employed. For example, visual features can be obtained through the intermediate layer output of a pre-trained classification network (such as ResNet, VGG, etc.), which is trained on a large image classification dataset, thus enabling the extraction of global semantic information from the image. Pixel features can be directly extracted from the original sample images through a simple convolutional layer stack or edge detection algorithms (such as Sobel, Canny, etc.) to capture local texture and edge details of the image. Simultaneously, the original sample images are input into a visual encoder to be trained. This encoder can be a randomly initialized convolutional neural network or a model that has been pre-trained on a general image task but has not yet been optimized for a specific generation task. After processing by this visual encoder, a basic feature representation can be obtained, which contains the preliminary encoded information of the original sample images.

[0043] Secondly, visual semantic features are constructed based on visual features and basic features, and pixel semantic features are constructed based on pixel features and basic features. After obtaining visual features, pixel features, and basic features, it is necessary to integrate this information at different levels to form more expressive semantic features. For example, visual semantic features can be constructed by simply concatenating visual features with basic features, or by mapping them to a common feature space using a multilayer perceptron. This approach allows visual semantic features to simultaneously contain high-level semantic information and basic encoded representations. Similarly, pixel semantic features can be constructed by concatenating pixel features with basic features, or by fusing them using another multilayer perceptron. In this way, pixel semantic features can preserve the detailed information of the image and, combined with basic features, provide fine guidance for subsequent reconstruction.

[0044] Next, the visual semantic features and pixel semantic features are fused, and the fused result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image. After constructing the visual semantic features and pixel semantic features, these two complementary features need to be integrated to fully utilize their respective advantages. For example, the fusion operation can be achieved by element-wise summation, averaging, or simple channel concatenation of the visual semantic features and pixel semantic features. The fused features contain both global semantic and local detail information of the image. Subsequently, the fused features are input into a pixel decoder. This pixel decoder can be a network composed of deconvolutional layers or a generative network based on upsampling and convolution operations. The pixel decoder is responsible for progressively upsampling the fused features and converting them into a reconstructed image of the same size as the original sample image.

[0045] Finally, the visual encoder to be trained is trained based on the original sample images and the reconstructed images to obtain the target visual encoder. To optimize the visual encoder to be trained so that it can simultaneously handle image understanding and generation tasks, a suitable training mechanism needs to be designed. For example, the training process can be based on a reconstruction loss function, which measures the pixel differences between the original sample image and the reconstructed image, such as mean squared error or mean absolute error. By minimizing this loss, the visual encoder to be trained is guided to learn how to encode image information so that the pixel decoder can accurately reconstruct the original image. During training, the parameters of the visual encoder to be trained are updated according to the gradient of the loss function until a preset number of training epochs or loss convergence condition is reached, ultimately yielding an optimized target visual encoder.

[0046] For example, the goal of an image processing system is to train a visual encoder that can accurately understand image content and generate high-quality images. Traditional methods often struggle to achieve both goals simultaneously. This embodiment proposes a novel training method to address this. Specifically, the system receives a raw sample image, such as a landscape photograph containing complex backgrounds and foreground objects. First, the system determines the visual and pixel features corresponding to this landscape photograph. Visual features can be extracted using an image classification model pre-trained on a large number of natural images; for example, this model identifies high-level semantic information such as "mountains," "rivers," and "trees" in the photograph. Simultaneously, pixel features can be extracted using a simple set of convolutional kernels, capturing low-level details such as "rock textures of mountains," "water ripple details of rivers," and "edge contours of leaves." Furthermore, this raw sample image is input into a visual encoder to be trained. This encoder may initially be a general feature extractor, whose output is called the base features.

[0047] Based on this, the system constructs visual semantic features according to the extracted visual and basic features. For example, by concatenating the feature vectors of visual features and basic features, a visual semantic representation containing global semantics and preliminary encoding information is formed. Simultaneously, the system constructs pixel semantic features based on pixel features and basic features. For example, by concatenating the feature vectors of pixel features and basic features, a pixel semantic representation containing local details and preliminary encoding information is formed. In this way, visual semantic features focus on the overall meaning of the image, while pixel semantic features focus on the fine structure of the image. The system can then fuse the constructed visual and pixel semantic features. For example, by concatenating these two types of features along the channel dimension, a more comprehensive fused feature is formed. This fused feature contains both high-level semantics and low-level detail information of the image. This fused feature is then input into a pixel decoder. The pixel decoder is responsible for progressively restoring the fused feature into an image, ultimately outputting a reconstructed image. This reconstructed image is generated by the system based on the features extracted by the encoder and the mapping relationship learned by the decoder, aiming to restore the original sample image as closely as possible.

[0048] Finally, the system trains the visual encoder to be trained based on the original sample image and the reconstructed image. For example, it calculates the pixel difference between the original sample image and the reconstructed image (e.g., calculating the mean absolute error of their pixel values) and uses this difference as part of the loss function. The parameters of the visual encoder to be trained are adjusted based on the loss value using a backpropagation algorithm. This training process continues until the visual encoder to be trained can generate reconstructed images that are highly similar to the original sample images, and its internal feature representation is optimized, ultimately yielding a target visual encoder. Through this training mechanism, the target visual encoder can not only effectively encode image information to support high-quality reconstruction, but also better preserve the semantic understanding capability of the image due to the fusion of high-level visual features and low-level pixel features during its training process.

[0049] In summary, the image processing method provided in this embodiment demonstrates a significant technical contribution in solving core problems in the field of image processing. Existing image processing methods face a dilemma when handling image generation and understanding tasks. For example, in the landscape photo example above, if route one is adopted, i.e., fine-tuning all parameters of the visual encoder to adapt to the generation task, although high reconstruction quality may be obtained, this fine-tuning process will destroy the general visual semantic representation originally learned by the encoder. This means that the fine-tuned encoder will significantly degrade in performance when recognizing high-level semantic information such as "mountains," "rivers," and "trees," resulting in catastrophic forgetting problems. On the other hand, if route two is adopted, i.e., completely freezing the pre-trained visual encoder and directly using its output features for image reconstruction, although the encoder's understanding ability can be preserved, the pre-trained visual encoder mainly focuses on high-level semantics and lacks attention to low-level pixel-level details such as "rock texture," "water ripple details," and "leaf outlines." Direct decoding will lead to severe artifacts and blurring in the reconstructed image. In the landscape photo example above, freezing the encoder may result in the reconstructed mountains lacking realistic rock texture, the river appearing blurry, and the leaf details of the trees being lost.

[0050] To address this challenge, the proposed technical solution employs an innovative training paradigm. Specifically, by determining the visual and pixel features corresponding to the original sample image and utilizing the basic features obtained from the visual encoder to be trained, both global semantics and local details of the image can be captured simultaneously. In the landscape photo example, this means that the encoder learns not only high-level concepts such as "mountains" and "rivers" during training, but also low-level information such as "rock texture" and "water ripple details." Subsequently, visual semantic features are constructed based on the visual and basic features, and pixel semantic features are constructed based on the pixel and basic features. This dual-stream construction method ensures that semantic and detail information can be processed independently and effectively. Furthermore, the visual and pixel semantic features are fused, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image. This step allows the decoder to simultaneously utilize the guidance of high-level semantics and the supplementation of low-level details, thereby generating a high-quality reconstructed image and significantly reducing artifacts and blur. In the landscape photo example, the reconstructed mountains will have clear rock textures, the rivers will exhibit natural water ripple details, and the leaves of the trees will be sharper. Finally, the target visual encoder is obtained by training the visual encoder under test based on the original sample images and the reconstructed images. This training mechanism allows the visual encoder under test to refine its internal feature representation while optimizing the reconstruction quality, thereby improving the generation ability without sacrificing the original understanding ability. Compared with the prior art, the technical solution of this embodiment avoids the catastrophic forgetting caused by full fine-tuning and solves the reconstruction quality problem caused by freezing the encoder. It achieves synergistic optimization of image understanding and generation tasks, providing a more robust and efficient foundation for the application of multimodal large language models in the field of image processing.

[0051] Furthermore, without specifying the feature extraction method, it may be impossible to effectively combine global semantics and detailed information, thereby affecting image reconstruction quality and model training performance. Therefore, in this embodiment, determining the visual features and pixel features corresponding to the original sample image includes: The original sample image is input to a visual encoder for encoding processing to obtain visual features; the original sample image is input to a pixel encoder for encoding processing to obtain pixel features; wherein, the visual encoder is a pre-trained visual encoder used to extract global semantic features of the original sample image; the pixel encoder is a pixel-level encoder used to extract detailed semantic features of the original sample image.

[0052] The original sample image is the raw image data to be processed. It can be a digital image of any format, such as JPEG or PNG, and serves as the input to the entire image processing flow. The visual encoder is a neural network model used to encode and process images, converting the input original sample image into visual features. This visual encoder can be a Transformer-based model, such as Vision Transformer (ViT) or SwinTransformer, which captures the global dependencies of the image through a self-attention mechanism; or it can be a Convolutional Neural Network (CNN)-based model, such as ResNet or EfficientNet, which progressively extracts image features through multiple layers of convolution and pooling operations. Visual features are the feature representations obtained after the visual encoder encodes the original sample image. These features are typically high-dimensional vectors or feature maps containing the semantic information of the image.

[0053] A pixel encoder is a neural network model used for encoding and processing images, transforming raw input images into pixel features. This pixel encoder can be the encoding part of a U-Net architecture, extracting local details of the image through downsampling and convolution operations; or it can be an encoder based on a Fully Convolutional Network (FCN), focusing on pixel-level feature extraction while preserving spatial resolution. Pixel features are the feature representations obtained after the pixel encoder encodes the raw sample image; these features typically contain low-level details such as texture, edges, and color.

[0054] Pre-trained visual encoders are a specific type of visual encoder that have been pre-trained on large-scale image datasets to learn rich, general visual semantic representations. For example, it could be a CLIP visual encoder pre-trained on the ImageNet dataset, capable of extracting high-level semantic concepts from images; or a ViT model pre-trained on the JFT-300M dataset, possessing powerful global semantic understanding capabilities. Global semantic features are features in an image that represent high-level concepts, overall structure, and main content, such as identifying object categories and scene types in an image. Pixel-level encoders are a specific type of pixel encoder designed to extract pixel-level detail information from images, rather than high-level semantics. For example, it could be a lightweight CNN specifically designed to extract image texture and edge information, such as learnable convolutional layers based on Sobel or Canny operators; or a convolutional network with a small receptive field, such as a multilayer perceptron (MLP) or a small ResNet block, focusing on feature extraction from local regions. Detail semantic features are features in an image that represent fine information such as low-level texture, edges, color, and local structure; these features are crucial for the quality of image reconstruction.

[0055] Based on this, a dual-stream encoding strategy was adopted when determining the visual and pixel features corresponding to the original sample images. Specifically, the original sample images were input in parallel to two independent encoders: a visual encoder and a pixel encoder. The visual encoder was configured as a pre-trained visual encoder, whose core function was to extract the global semantic features of the original sample images. This pre-trained characteristic enabled the visual encoder to provide high-level semantic representations learned from large-scale data, serving as "anchors" for semantic capabilities in subsequent model training, effectively preventing catastrophic forgetting during training, and thus maintaining its strong performance in image understanding tasks. Simultaneously, the original sample images were also input to the pixel encoder, which was configured as a pixel-level encoder, whose main responsibility was to extract detailed semantic features from the original sample images. The pixel-level encoder focused on capturing fine information such as low-level texture, edges, and colors of the image, compensating for the low-level visual signals that the pre-trained visual encoder might lose due to its focus on high-level semantics. In this way, this scheme achieved effective separation and parallel acquisition of high-level global semantic information and low-level pixel detail information in the initial stage of feature extraction. The visual encoder and pixel encoder, acting as two frozen teacher models, provide complementary supervision signals: high-level semantic representation and low-level pixel detail features, respectively. This dual-stream feature extraction mechanism enables the subsequent construction of visual and pixel semantic features to fully utilize information at these two different granularities, laying a solid foundation for high-quality image reconstruction and model training. This overcomes the problems in existing technologies, such as artifact blurring caused by freezing the visual encoder and decreased understanding ability due to fine-tuning the visual encoder.

[0056] In practical applications, two completely frozen teacher models can be introduced to provide complementary supervision signals. Semantic Teacher (Visual Encoder) A frozen, pre-trained visual encoder provides high-level semantic representations that have been pre-trained on a large scale, serving as "anchors" for the student's network semantic capabilities and preventing them from forgetting their original understanding during training. Pixel Teacher The (pixel encoder) uses a frozen pixel-level encoder to provide low-level detail features such as texture, edges, and color, compensating for the low-level visual signals lost by the visual encoder due to its focus on high-level semantics. It enables the semantic teacher to extract features from a given input image x. Pixel teacher feature extraction These two features represent the high-level semantic information and low-level pixel detail information of the image, respectively, and will serve as the distillation targets for the student network's two-stream processing. Furthermore, the student network uses a trainable visual encoder as the training target. As the core. Given an input image. First, the basic features are extracted using this visual encoder. This feature was then split into two parallel processing paths for further processing.

[0057] For example, the following approach can be used to determine the visual and pixel features corresponding to the original sample image. First, the original sample image, such as a color image with a resolution of 256x256 pixels, is input into a pre-trained visual encoder. This visual encoder can be a model based on the ViT-B / 16 architecture, which has been pre-trained on large-scale datasets such as ImageNet-21K and LAION-400M with its parameters frozen. This visual encoder processes the image patch sequence through its Transformer layer, outputting a visual feature vector containing global semantic information of the image, such as a 768-dimensional feature vector. Simultaneously, the same original sample image is input into a pixel encoder. This pixel encoder can be a lightweight U-Net encoding part composed of multiple convolutional layers and residual blocks, designed to capture the local texture and edge information of the image, for example, outputting a feature map with a resolution of 64x64 and 128 channels as pixel features. These two encoders work in parallel, independently extracting complementary visual and pixel features from the original sample image, representing the high-level semantics and low-level details of the image, respectively.

[0058] In summary, this approach effectively addresses the challenge of simultaneously capturing global semantics and detailed information in existing technologies when determining the features of original sample images. Specifically, by introducing a pre-trained visual encoder to extract global semantic features, the rich semantic knowledge learned from large-scale data can be fully utilized, ensuring the accuracy and completeness of high-level semantic information. This avoids catastrophic forgetting that might occur during encoder fine-tuning, thus maintaining the model's performance in image understanding tasks. Furthermore, by introducing a pixel-level encoder to extract detailed semantic features, fine information such as low-level texture, edges, and color can be effectively captured, compensating for the loss of detailed information during compression in traditional visual encoders. This significantly improves the quality of reconstructed images and reduces artifacts and blurring. This dual-stream parallel extraction mechanism allows subsequent image processing to simultaneously obtain high-quality global semantics and local detailed information, providing a solid foundation for constructing more accurate visual semantic features and pixel semantic features, thereby enhancing the realism of image reconstruction and the overall performance of the model.

[0059] Furthermore, how to efficiently and accurately extract key semantic information from basic features to avoid information loss and computational redundancy, thereby improving the accuracy and efficiency of feature fusion, is a technical challenge that needs to be addressed. In this embodiment, constructing visual semantic features based on the visual features and the basic features includes: The basic features are input into the semantic encoding module for semantic extraction to obtain semantic features; visual semantic features are constructed based on the visual features and the semantic features.

[0060] The semantic encoding module can be a neural network module specifically designed to extract high-level semantic information from input features. For example, this module can be composed of multiple stacked convolutional layers, batch normalization layers, and activation function layers to capture semantic information at different scales; alternatively, it can employ an attention-based structure, using self-attention to identify and aggregate key semantic elements from the basic features. Semantic extraction refers to the process of identifying, extracting, and representing the inherent meaning or high-level concepts from the basic features. Specifically, feature mapping and abstraction can be performed through convolutional neural network (CNN) layers to gradually transform low-level features into high-level semantic features; or, sequence models such as recurrent neural networks (RNNs) or Transformers can be used to process feature sequences, capturing the contextual dependencies between features to extract richer semantic information. Constructing visual semantic features refers to effectively combining the visual features and the semantic features to form a more comprehensive and expressive feature representation. For example, the visual features and semantic features can be merged along the channel dimension by feature concatenation, and then further feature fusion and dimensionality reduction can be performed through a fully connected layer or a convolutional layer; or, an attention mechanism can be used to weight the visual features according to the importance of the semantic features, thereby achieving deeper interaction and fusion.

[0061] Based on this, a semantic encoding module is introduced to extract semantic information from basic features and combine it with visual features to construct visual semantic features. This aims to solve the problem of how to efficiently and accurately extract key semantic information from basic features during image reconstruction. Specifically, after determining the visual and pixel features corresponding to the original sample image and processing the original sample image using a visual encoder to be trained to obtain basic features, these basic features are first input into the semantic encoding module. This semantic encoding module is specifically responsible for extracting pure and focused semantic information from the basic features, effectively filtering out possible redundancy or noise interference, ensuring that the obtained semantic features are more representative. Subsequently, the refined semantic features are combined with the original visual features. Since visual features usually come from a pre-trained visual encoder, which is good at capturing the global semantic information of an image, while the newly extracted semantic features focus more on specific semantic details mined from the basic features. By organically combining the two, the semantic information can be complemented and enhanced, thereby constructing more comprehensive, robust, and information-rich visual semantic features. This construction method not only avoids the information loss and computational redundancy that may result from directly using the original basic features, but also provides high-quality input for the subsequent steps of fusing visual semantic features and pixel semantic features, and using a pixel decoder to decode the fusion result to obtain the reconstructed image corresponding to the original sample image. This significantly improves the accuracy and efficiency of feature fusion, thereby optimizing the quality of image reconstruction.

[0062] For example, the semantic encoding module can be a network structure consisting of three convolutional layers (e.g., each convolutional layer followed by batch normalization and ReLU activation) and a global average pooling layer. The first convolutional layer compresses the number of channels in the basic features, for example, reducing it from 1024 dimensions to 512 dimensions. Subsequent convolutional layers further extract high-level semantic information, and the global average pooling layer aggregates spatial information into a fixed-dimensional semantic vector, for example, 256 dimensions. During the semantic extraction process, feature mapping operations are performed by sliding convolutional kernels across the basic features. For example, a 3x3 convolutional kernel with a stride of 1 and padding of 1 is used to maintain the feature map size, and the number of channels is gradually increased to capture more complex semantic patterns. When constructing visual semantic features based on the visual features and the semantic features, the visual features obtained from the pre-trained visual encoder (e.g., a 256-dimensional vector) and the semantic features obtained from the semantic encoding module (e.g., a 256-dimensional vector) can be concatenated by channels to form a 512-dimensional fused feature vector. Subsequently, the 512-dimensional fused feature vector can be input into a visual dimensionality reduction module consisting of two fully connected layers. The first fully connected layer maps the dimensions to 256 dimensions, and the second fully connected layer outputs the final visual semantic features.

[0063] In summary, the above processing significantly improves the accuracy and efficiency of feature fusion, providing high-quality semantic input for subsequent image reconstruction, thereby helping to generate reconstructed images with richer details, fewer artifacts, and higher visual quality. Based on this, features may suffer from semantic incompatibility or loss of detail due to a lack of alignment mechanisms, thus affecting the accuracy and clarity of the reconstructed image. Therefore, in this embodiment, constructing visual semantic features based on the visual features and the semantic features includes: The semantic features are distilled and aligned with the visual features, and the distilled and aligned semantic features are then dimensionality-reduced using a visual dimensionality reduction module to obtain visual semantic features; wherein, the step of constructing pixel semantic features based on the pixel features and the basic features includes: performing dimensionality reduction on the basic features using a pixel dimensionality reduction module to obtain initial pixel semantic features; and distilling and aligning the initial pixel semantic features with the pixel features to obtain pixel semantic features.

[0064] The process of distilling and aligning the semantic features with the visual features aims to ensure consistency or compatibility in the semantic space between the semantic features obtained by the semantic encoding module and the visual features obtained by the pre-trained visual encoder. This helps bridge the semantic differences that may arise from different feature extraction paths, making subsequent feature fusion more effective. Implementation methods can include: one approach is to use a loss function-based method, such as calculating the distance between semantic and visual features (e.g., L2 distance, cosine similarity) as the distillation loss and incorporating it into the overall training objective to guide the semantic features towards alignment with the visual features; another approach is to utilize attention mechanisms or transformer structures to learn a mapping function to project the semantic feature space onto the visual feature space, thereby achieving soft alignment of features. A visual dimensionality reduction module then performs dimensionality reduction on the distilled and aligned semantic features. This module aims to reduce the dimensionality of the features while preserving their core semantic information as much as possible. While high-dimensional features contain rich information, they may also introduce redundancy and increase computational burden; dimensionality reduction helps improve the model's efficiency and generalization ability. Implementation methods can include: one approach is to use linear dimensionality reduction methods such as Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA) to project features by finding the main directions of change in the data; another approach is to use non-linear dimensionality reduction methods such as pooling layers or 1x1 convolutional layers in autoencoders or convolutional neural networks (CNNs) to compress feature representations by learning an encoder-decoder structure. Visual semantic features are obtained, which are semantic features after distillation alignment and dimensionality reduction. They fuse high-level semantic information and are optimized into a more compact and discriminative representation. These features aim to capture the global semantic content of the image while maintaining consistency with the features extracted by the visual encoder, providing high-quality semantic input for subsequent feature fusion.

[0065] The basic features are dimensionality reduced using a pixel dimensionality reduction module. This module reduces the dimensionality of the basic features to extract feature representations focused on low-level details. Basic features may contain rich but redundant information; dimensionality reduction removes this redundancy, focusing on pixel-level texture, edge, and other details, providing refined input for subsequent pixel semantic feature construction. Implementation methods include: one approach is to use a combination of convolutional and pooling layers, for example, by using a series of stride convolutions and max pooling operations to gradually reduce the size and number of channels of the feature map; another approach is to use a learnable linear projection layer to map high-dimensional basic features to a lower-dimensional space. Initial pixel semantic features are obtained, which are feature representations obtained after processing the basic features using the pixel dimensionality reduction module. These features primarily carry low-level image details, such as texture, edges, and color gradients, and have relatively low dimensionality, preparing for subsequent distillation alignment with pixel features. The initial pixel semantic features are then distilled and aligned with the pixel features. This distillation alignment ensures that the initial pixel semantic features (derived from the basic features and dimensionality reduced) maintain consistency with the pixel features extracted by the pixel encoder in terms of pixel detail dimension. This helps compensate for low-level details that the visual encoder may lose during compression, ensuring the sharpness and realism of the reconstructed image. Implementation methods can include: one approach is to use a feature map matching-based loss function, such as calculating the L1 or L2 distance between the initial pixel semantic features and the pixel features in the spatial dimension, to force them to maintain similarity at the level of detail; another approach is to utilize an adversarial learning framework, where a discriminator attempts to distinguish between features before and after alignment, while a generator strives to generate initial pixel semantic features aligned with the pixel features. The resulting pixel semantic features are the initial pixel semantic features after distillation and alignment. They effectively integrate pixel-level detail information of the image and maintain a high degree of consistency with the features extracted by the pixel encoder. This feature focuses on local textures, edges, and other details of the image, providing high-quality detailed input for subsequent feature fusion, thereby significantly improving the sharpness and realism of the reconstructed image.

[0066] Based on this, this solution addresses the problem of semantic information incompatibility or loss of detail information caused by the lack of an effective feature alignment mechanism during image reconstruction. To this end, when constructing visual semantic features, the semantic features obtained by the semantic encoding module are first distilled and aligned with the visual features obtained by the pre-trained visual encoder. This step ensures the consistency of high-level semantic information across different feature streams, avoiding semantic conflicts caused by differences in feature sources. Subsequently, the distilled and aligned semantic features are dimensionality reduced using a visual dimensionality reduction module. This effectively compresses feature dimensions while preserving key semantic information, thereby optimizing the efficiency and quality of feature representation and providing refined semantic input for subsequent fusion. Simultaneously, when constructing pixel semantic features, this embodiment uses a pixel dimensionality reduction module to reduce the dimensionality of the basic features obtained by the visual encoder processing the original sample image to obtain initial pixel semantic features. This process focuses on extracting and refining the low-level detail information of the image, such as texture and edges, from the basic features. Then, this initial pixel semantic feature is distilled and aligned with the pixel features extracted by the pixel encoder. This alignment mechanism enables precise integration of pixel-level details and dimensionality-reduced features at the pixel level, ensuring that low-level details such as image texture and edges are effectively utilized and preserved. Through the two parallel feature construction processes described above, this embodiment effectively solves the problems of semantic information incompatibility and detail loss during feature fusion. The visual semantic feature stream focuses on the alignment and compression of high-level semantics, while the pixel semantic feature stream focuses on the extraction and integration of low-level details. The synergistic effect of these two feature streams allows for a fusion result that contains both rich semantic information and retains fine pixel details in the subsequent feature fusion stage, thereby significantly improving the accuracy and clarity of the reconstructed image and avoiding the blurring or artifact problems common in traditional methods.

[0067] In practice, the semantic stream corresponding to the visual encoder aims to learn and retain high-level semantic information. The output features of the visual encoder to be trained first pass through the deep semantic encoding module. Further abstract semantics are extracted to obtain semantic features. This feature will be combined with the semantic teacher's output. Distillation alignment is performed. Then, it is processed through a dimensionality reduction module. Compress semantic features into low-dimensional semantic latent variables .

[0068] The pixel stream corresponding to the pixel encoder aims to supplement the low-level detail information lost by the visual encoder. The output features (basic features) of the visual encoder to be trained skip the semantic encoding module and are directly processed by the dimensionality reduction module. Generate pixel latent variables that focus on low-level details. This latent variable will be related to the encoded output of the pixel teacher. Distillation alignment is performed. This design allows the pixel stream to retain more original detail information, avoiding the loss of detail caused by further abstraction of the semantic encoding module.

[0069] For example, when constructing visual semantic features, the following steps can be taken: First, the semantic features output by the semantic encoding module are distilled and aligned with the visual features extracted by a pre-trained visual encoder (e.g., a CLIP-based visual encoder). This can be achieved by defining a distillation loss function, which can be the L2 distance between the semantic and visual features in the feature space, or by calculating their similarity after mapping them to the same space using a multilayer perceptron (MLP). For example, a semantic teacher model can be set up whose output is... Then semantic features and Alignment is then performed. Next, the distilled and aligned semantic features are dimensionality reduced using a visual dimensionality reduction module. This visual dimensionality reduction module can be a neural network composed of multiple convolutional and pooling layers, or an autoencoder with a bottleneck structure. Its function is to compress high-dimensional semantic features into low-dimensional visual semantic features. For example, the feature dimension can be reduced from 2048 to 512.

[0070] Meanwhile, when constructing pixel semantic features, the following steps can be adopted: First, the basic features obtained by the visual encoder to be trained from the original sample image are reduced in dimensionality using a pixel dimensionality reduction module to obtain initial pixel semantic features. This pixel dimensionality reduction module can be a lightweight convolutional network, for example, containing several convolutional layers with a stride of 2, used to progressively reduce the spatial size and number of channels of the feature map, thereby extracting feature representations focused on local details from the basic features. For example, the basic feature h is processed by the dimensionality reduction module... Generate pixel latent variables Subsequently, the initial pixel semantic features are distilled and aligned with the pixel features extracted by a pixel encoder (e.g., a U-Net structure specifically designed for extracting local texture and edge information from an image). This can be achieved through another distillation loss function, which could be the L1 distance between the initial pixel semantic features and the pixel features in the pixel dimension, or through a discriminator to determine whether they come from the same distribution, thus forcing them to maintain consistency at the level of detail. For example, a pixel teacher model could be set up whose encoded output is... Then and Align them.

[0071] In summary, by introducing distillation alignment and dimensionality reduction mechanisms, semantic conflicts are avoided, while the visual dimensionality reduction module improves feature representation efficiency while preserving key semantic information. Simultaneously, the construction of pixel semantic features, through pixel dimensionality reduction and distillation alignment, effectively extracts and integrates low-level image details from basic features, compensating for details that may be lost during compression by the visual encoder. This dual-channel feature construction strategy ensures that the final fused features contain rich global semantic information while preserving fine local pixel details, thus significantly improving the accuracy and clarity of the reconstructed image and effectively avoiding common artifacts and blurring phenomena in reconstructed images. Furthermore, simple fusion methods may fail to fully integrate high-level semantic information and low-level detail information, leading to conflicts or mismatches in the fused features, thereby affecting the quality of the reconstructed image, such as artifacts, blurring, or loss of key details. To address this, in this embodiment, the process of fusing the visual semantic features and the pixel semantic features, and then decoding the fusion result using a pixel decoder to obtain the reconstructed image corresponding to the original sample image, includes: The visual semantic features and the pixel semantic features are concatenated, and the concatenated semantic features are mapped using a fusion module to obtain the target semantic features. The target semantic features are then input into a pixel decoder for decoding to obtain the reconstructed image corresponding to the original sample image.

[0072] The concatenation of visual semantic features and pixel semantic features refers to linking features from two different sources along a certain dimension to form a longer feature representation containing more information. This operation aims to fully preserve the high-level semantic information represented by the visual semantic features and the low-level detailed information represented by the pixel semantic features, avoiding any information loss in the initial stage of fusion, thus providing a comprehensive input foundation for subsequent feature processing. Specifically, the concatenation operation can be performed along the channel dimension, for example, by superimposing the channel numbers of the visual semantic feature map and the pixel semantic feature map to form a feature map with richer channel information; or it can be performed along the spatial dimension, for example, by connecting the two feature maps along the height or width direction.

[0073] Correspondingly, the mapping processing of the concatenated semantic features using a fusion module refers to the nonlinear transformation and integration of the concatenated features through a specially designed neural network module. The role of this fusion module is to adaptively learn how to effectively combine high-level semantic information and low-level detailed features, eliminating potential redundancy or inconsistencies between the two, thereby generating a unified and more information-rich target semantic feature. This fusion module can consist of one or more fully connected layers, supplemented by activation functions (such as ReLU, GELU) for nonlinear processing; or it can be a convolutional neural network structure containing convolutional layers, batch normalization layers, and activation functions to process input in the form of feature maps; or it can employ attention mechanisms, such as self-attention or cross-attention, to dynamically weight the importance of different features.

[0074] Correspondingly, obtaining target semantic features refers to the final feature representation that integrates visual semantics and pixel semantics information, obtained after processing by the fusion module. These target semantic features serve as input to the pixel decoder, and their quality directly affects the final reconstructed image.

[0075] Correspondingly, inputting the target semantic features into the pixel decoder for decoding means taking the target semantic features output by the fusion module as input, and progressively upsampling and converting them into an image in pixel space through the pixel decoder. The pixel decoder is a neural network model whose core function is to reconstruct the pixel details and structure of the image based on the input feature representation. This pixel decoder can consist of a series of upsampling layers (e.g., transposed convolutional layers, bilinear interpolation followed by convolutional layers) and convolutional layers to progressively restore the spatial resolution and details of the image; alternatively, it can employ the decoding part of the U-Net architecture, introducing features from the encoder through skip connections to better preserve detailed information.

[0076] Accordingly, obtaining the reconstructed image corresponding to the original sample image refers to the image finally output by the pixel decoder, which should be as similar as possible to the original sample image at the pixel level. The quality of the reconstructed image is a key indicator for evaluating the effectiveness of the entire image processing method and serves as a supervision signal for the visual encoder to be trained during the training process.

[0077] Based on this, by concatenating visual semantic features and pixel semantic features, the integrity of both features is maintained and directly superimposed in the initial stage, avoiding information loss or conflicts caused by simplified fusion, and providing a comprehensive input foundation for subsequent processing. On this basis, a fusion module is used to map the concatenated semantic features, adaptively adjusting and integrating high-level semantics and low-level detail features through nonlinear transformations, eliminating redundancy and inconsistencies between features, and generating more unified and richer target semantic features. Finally, the target semantic features are input to the pixel decoder for decoding, enabling the decoder to more accurately reconstruct the image structure and restore details based on the optimized fused features. This refined feature fusion mechanism fully utilizes the complementary advantages of global semantics and local details, achieving high-quality image reconstruction without sacrificing image understanding capabilities.

[0078] In practice, the semantic latent variables output by the semantic stream will be... and pixel latent variables of pixel stream output The data is stitched together along the channel dimension and then processed through a fusion module. The mapping to the final latent variable is accomplished through the following formula (1): (1) The fusion module dynamically combines semantic and pixel detail information through learnable parameters, enabling the final latent variables to... It contains both high-level semantic structure and low-level pixel details. Latent variables Input to pixel decoder In the process, a reconstructed image is generated. .

[0079] For example, first, the semantic latent variable z output by the semantic stream... sem and pixel latent variables of pixel stream output The features are concatenated along the channel dimension to form a longer-dimensional feature. For example, if z sem The number of channels is C1. If the number of channels is C2, then the number of feature channels after concatenation is C1 + C2. Next, the concatenated features are input into a fusion module F. This fusion module F can be implemented as a convolutional neural network consisting of multiple residual blocks. Each residual block contains two convolutional layers, a batch normalization layer, and an activation function (such as GELU), and the input is directly added to the output through skip connections. The fusion module F performs non-linear mapping processing through its internal convolutional layers and activation functions, learning how to extract and integrate useful information from the concatenated features, thus mapping it to the final latent variable Z. The latent variable Z contains both high-level semantic structure and low-level pixel details. Finally, the latent variable Z is input into the pixel decoder G, which can be a decoder part based on a U-Net architecture, containing a series of upsampling layers (e.g., using transposed convolution or nearest neighbor interpolation followed by convolution) and convolutional layers, progressively decoding the latent variable Z into the reconstructed image. .

[0080] In summary, the above processing significantly improves the clarity and realism of the reconstructed image, reduces artifacts and blurring, and effectively restores key details, thus obtaining a high-quality reconstructed image.

[0081] Furthermore, relying solely on a single loss function may fail to fully optimize the encoder, leading to poor reconstructed image quality or reduced understanding capabilities. Therefore, in this embodiment, training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder includes: The pixel difference loss value of the original sample image and the reconstructed image in the pixel difference dimension is determined according to the reconstruction loss function; the feature loss value of the original sample image and the reconstructed image in the feature dimension is determined according to the perceptual loss function; the semantic loss value of the intermediate features in the semantic stream of the original sample image and the visual features in the semantic dimension is determined according to the semantic distillation loss function; the pixel loss value of the intermediate features in the pixel stream of the original sample image and the pixel features in the pixel dimension is determined according to the pixel distillation loss function; the generation loss value of the original sample image and the reconstructed image in the image generation dimension is determined according to the adversarial loss function; the visual encoder to be trained is trained based on the pixel difference loss value, the feature loss value, the semantic loss value, the pixel loss value, and the generation loss value until a target visual encoder that meets the training stopping condition is obtained.

[0082] The reconstruction loss function is a mathematical function used to quantify the pixel-level differences between the original and reconstructed images. Its role is to directly measure the accuracy of the reconstructed image at the pixel level, ensuring that the reconstructed image can restore the details and structure of the original image as much as possible. The pixel difference dimension refers to comparing the differences between the original and reconstructed images pixel by pixel in the image's pixel space. The pixel difference loss value is calculated using the reconstruction loss function and reflects the quantification result of this pixel-level difference. Common implementations include L1 loss (mean absolute error) and L2 loss (mean squared error). L2 loss penalizes larger errors by calculating the sum of squared differences in corresponding pixel values, while L1 loss calculates the sum of absolute differences and is less sensitive to outliers.

[0083] Correspondingly, the perceptual loss function is a function that measures image similarity by comparing the feature representations extracted from images in pre-trained feature extraction networks (such as VGG, ResNet, etc.). Its role is to capture the high-level semantic and structural information of images, making the reconstructed images visually more natural and consistent with human perception. Feature dimension refers to the feature vector space corresponding to the image at different layers of the pre-trained feature extraction network. The feature loss value is calculated using the perceptual loss function, reflecting the differences between the original image and the reconstructed image in these feature dimensions. For example, a pre-trained VGG network can be used to extract feature maps of an image, and then the L1 or L2 distance between the feature maps output by a specific layer of the VGG network in the original image and the reconstructed image can be calculated as the perceptual loss.

[0084] Correspondingly, the semantic distillation loss function aims to enable the visual encoder to maintain its original semantic understanding ability during training through knowledge distillation. Intermediate features in the semantic stream refer to the feature representations of the original sample image after passing through a semantic processing module (such as a pre-trained semantic encoder or teacher model), representing its high-level semantic information. Visual features refer to the global semantic features extracted by the pre-trained visual encoder (usually frozen) for understanding the task. Semantic dimension refers to the feature space in which these semantic features reside. The semantic loss value is calculated using the semantic distillation loss function and measures the alignment of the semantic features generated by the visual encoder to be trained with the pre-defined visual features (or the features output by the semantic teacher model) in the semantic dimension. For example, a frozen pre-trained visual encoder can be set as the semantic teacher, and the L1 or L2 distance between the features output by the visual encoder to be trained and the features output by the semantic teacher can be calculated as the semantic distillation loss.

[0085] Correspondingly, the pixel distillation loss function aims to enhance the learning of pixel-level detail information by the visual encoder under training through knowledge distillation, thereby improving the quality of the reconstructed image. Intermediate features in the pixel stream refer to the feature representations of the underlying detail information generated after the original sample image has passed through a pixel processing module (such as a pre-trained pixel-level encoder or teacher model). Pixel features are the features extracted by the pixel encoder (usually frozen) to capture the semantic details of the image. Pixel dimension refers to the feature space in which these pixel features reside. The pixel loss value is calculated using the pixel distillation loss function and is used to measure the alignment between the pixel-level features generated by the visual encoder under training and the preset pixel features (or the features output by the pixel teacher model) in the pixel dimension. For example, a frozen pre-trained pixel encoder can be set as the pixel teacher, and the L1 or L2 distance between the pixel-level features output by the visual encoder under training and the features output by the pixel teacher can be calculated as the pixel distillation loss. Interpolation alignment of the features may also be necessary.

[0086] Correspondingly, the adversarial loss function is a loss function designed based on the idea of ​​Generative Adversarial Networks (GANs) to improve the realism and texture details of generated images. Its role is to introduce a discriminator network, enabling the generator (in this case, the visual encoder to be trained and its subsequent decoding part) to generate images that are difficult for the discriminator to distinguish from real images. The image generation dimension refers to the difference in visual realism and texture details between the generated image and the real image in image space. The generation loss value is calculated using the adversarial loss function, reflecting the generator's ability to deceive the discriminator. Common implementations include standard GAN loss, WGAN loss, Hinge loss, etc. Among them, Hinge loss performs well in terms of training stability, optimizing the discriminator by maximizing the discrimination score of the real image and minimizing the discrimination score of the generated image, while simultaneously optimizing the generator by minimizing the discrimination score of the generated image.

[0087] Correspondingly, training stopping conditions are criteria used to determine whether model training has achieved the expected results or reached a convergence state. Their purpose is to avoid overfitting or underfitting the model and to conserve computational resources. Common training stopping conditions include: reaching a preset maximum number of training epochs, the validation set loss no longer decreasing for several consecutive epochs, the validation set accuracy reaching a preset threshold, or the training loss falling below a certain minimum value. The target visual encoder refers to the visual encoder to be trained that meets the training stopping conditions after joint training using the above multiple loss functions. It possesses the ability to support high-quality image generation while maintaining its original understanding capabilities.

[0088] Based on this, a multi-dimensional loss function is introduced to comprehensively train the visual encoder to address the problem that a single loss function cannot fully optimize the encoder, leading to low-quality reconstructed images or reduced semantic understanding. Specifically, the entire training process revolves around optimizing the visual encoder to be trained, enabling it to extract fundamental features from the original sample images that support high-quality image reconstruction while maintaining the original semantic understanding capabilities. During training, the system calculates the pixel difference loss value between the original sample image and the reconstructed image in the pixel difference dimension based on the reconstruction loss function. This loss value directly affects the pixel level, ensuring that the reconstructed image is highly consistent with the original image in terms of detail and structure, thereby effectively reducing artifacts and blurring. Simultaneously, a perceptual loss function is used to calculate the feature loss value between the original sample image and the reconstructed image in the feature dimension. This loss value compares high-level semantic features, making the reconstructed image visually more natural and consistent with human perception, avoiding visual inconsistencies that may result from focusing solely on pixel differences. To address the potential decline in understanding capabilities that may result from fine-tuning the visual encoder, this scheme introduces a semantic distillation loss function and a pixel distillation loss function. The semantic distillation loss function constrains the intermediate features in the semantic stream generated by the visual encoder under training to maintain consistency with the preset visual features in the semantic dimension. This effectively protects the original global semantic understanding ability of the visual encoder while training the generation task, avoiding catastrophic forgetting. Similarly, the pixel distillation loss function constrains the intermediate features in the pixel stream generated by the visual encoder under training to maintain consistency with the preset pixel features in the pixel dimension. This strengthens the encoder's ability to extract details such as underlying textures and edges, further improving the precision of the reconstructed image. Furthermore, to further enhance the realism and generation stability of the reconstructed image, this scheme also introduces an adversarial loss function. This loss function, through an adversarial training mechanism, makes it difficult for the visual encoder under training and its subsequent decoding part to distinguish between real and fake images, thereby improving the visual quality and realism of the reconstructed image in the image generation dimension. Finally, the system jointly optimizes the pixel difference loss value, feature loss value, semantic loss value, pixel loss value, and generation loss value, and iteratively trains the visual encoder under training. This comprehensive use of multiple loss functions enables the model to achieve balanced optimization across multiple dimensions, including pixel-level accuracy, perceptual quality, semantic consistency, pixel detail, and image realism. By continuously training until a preset training stopping condition is met, a target visual encoder is ultimately obtained that can both support high-quality image generation and effectively maintain the original image understanding capabilities. This training strategy is closely integrated with the image processing workflow, providing a high-quality coding foundation for subsequent image reconstruction and potential image editing tasks, ensuring the system's superior performance in both generation and understanding tasks.

[0089] In practice, the model is trained end-to-end by minimizing the joint loss function, which can be achieved by the following formula (2): (2) in , , , , These are the weighting coefficients for each loss term, used to balance the contribution of different losses to model optimization. The values ​​of each weighting coefficient are adjusted according to the actual task requirements to achieve an optimal balance between maintaining comprehension ability and improving generation quality.

[0090] Reconstruction losses Calculate the pixel-level differences between the input image and the reconstructed image using L2 loss: , where N is the total number of pixels in the image. This loss directly constrains the reconstructed image to remain consistent with the input at the pixel level.

[0091] Perceived loss Features are extracted using a pre-trained convolutional neural network, and the differences between the input and reconstructed images in the feature space are calculated: ,in Indicates the network number The feature extraction function of the layer. Perceptual loss focuses on the high-level structure and semantic content of the image, rather than pixel-by-pixel alignment, and can generate images that are more natural and more in line with human visual perception.

[0092] Semantic distillation loss Constrain the difference between intermediate features of the semantic stream and the output features of the semantic teacher: ,in The feature dimension is denoted by . This loss forces the semantic encoding module output of the student network to be consistent with the frozen semantic teacher, thereby ensuring that the student network does not lose its original semantic understanding ability during training.

[0093] Pixel distillation loss The difference between constrained pixel stream output features and pixel teacher-encoded output: This loss forces the pixel stream to learn the low-level texture and detail features captured by the pixel teacher, compensating for the low-level visual signals lost by the visual encoder due to its focus on high-level semantics. Since the spatial resolutions of the pixel teacher and student networks may be inconsistent, the features of the pixel teacher need to be interpolated and aligned before computation.

[0094] Combating losses By introducing a discriminator network and employing adversarial training, the realism of the generated images is improved. A Hinge loss form is used: , .in This is the discriminator network. Adversarial loss can improve the detail quality and realism of the generated images, making the reconstructed images sharper and the textures more natural. To ensure training stability, the discriminator is kept off during the initial training phase, and adversarial training is enabled only after the reconstruction quality reaches a certain level.

[0095] Furthermore, to balance the retention of understanding capabilities with the learning of generative capabilities, a differentiated learning rate strategy can be employed. This applies to the visual encoder being trained. Use a lower learning rate (in The normal base learning rate is applied to other modules. This protective fine-tuning strategy results in smaller parameter updates and smoother changes in feature distribution for the visual encoder, thus maximizing the protection of pre-trained weights while allowing them to adapt to the generation task and avoiding a collapse in understanding ability due to drastic shifts.

[0096] For example, suppose the visual encoder to be trained is a Transformer-based encoder, and the pixel decoder is a U-Net-based decoder. Before training begins, the parameters of both the visual encoder and pixel decoder are initialized. Simultaneously, a frozen VGG-19 network can be pre-programmed as a feature extractor for perceptual loss, and its output feature maps are used to calculate the perceptual loss. For semantic distillation, a pre-trained and frozen CLIP visual encoder can be used as a semantic teacher, and its output features are used as visual features. For pixel distillation, a pre-trained and frozen ResNet-50 network can be used as a pixel teacher, and its output features are used as pixel features. Furthermore, a discriminator network, such as a PatchGAN discriminator, is needed for adversarial training. During training, each iteration performs the following operations: First, forward propagation is performed, inputting the original sample image into the visual encoder to obtain basic features. Then, following the steps described in the image processing method, visual semantic features and pixel semantic features are constructed, fused, and then used by the pixel decoder to generate a reconstructed image. Next, loss calculation is performed; the pixel difference loss value can use the L2 loss function to calculate the mean squared error between the original sample image and the reconstructed image. The feature loss value can be calculated by inputting the original sample image and the reconstructed image into a frozen VGG-19 network, extracting feature maps from layers `relu1_1`, `relu2_1`, `relu3_1`, `relu4_1`, and `relu5_1`, and then calculating the sum of the L1 distances between these feature maps. The semantic loss value can be calculated by inputting the original sample image into a frozen CLIP visual encoder to obtain visual features. Simultaneously, intermediate features from the semantic stream of the visual encoder being trained during the processing of the original sample image are extracted. Then, the cosine similarity loss or L2 distance between these two feature vectors is calculated. The pixel loss value can be calculated by inputting the original sample image into a frozen ResNet-50 network to obtain pixel features. Simultaneously, intermediate features from the pixel stream of the visual encoder being trained during the processing of the original sample image are extracted. The L2 distance between these two features is calculated. The generation loss value can be calculated by inputting the reconstructed image into the discriminator, calculating the probability of it being classified as a real image, and calculating the generator's loss according to the Hinge loss function. Simultaneously, the discriminator is trained to distinguish between real and reconstructed images. Finally, backpropagation and optimization are performed to calculate the total loss. The Adam optimizer is then used to backpropagate the total loss and update the parameters of the visual encoder and pixel decoder to be trained. The discriminator is optimized separately, and its parameters are updated using the Adam optimizer. In terms of optimizer settings, a lower learning rate can be used for the visual encoder, such as 0.1 times the base learning rate, while the base learning rate can be used for the pixel decoder and discriminator.The training process continues until the reconstruction quality metrics (such as PSNR or SSIM) on the validation set no longer improve for 10 consecutive epochs, or until the preset maximum number of training epochs (e.g., 100 epochs) is reached. Through the specific training process described above, image reconstruction quality and the understanding ability of the visual encoder can be effectively balanced, ultimately resulting in a high-performance target visual encoder.

[0097] In summary, by introducing a multi-dimensional loss function to train the visual encoder, this approach effectively solves the technical problem in traditional methods where a single loss function cannot comprehensively optimize the encoder, leading to low-quality reconstructed images or reduced understanding capabilities. Specifically, the pixel difference loss ensures high fidelity at the pixel level, reducing blur and artifacts; the perceptual loss makes the reconstructed image visually more natural and conforms to human perception habits; the semantic distillation loss effectively protects the original global semantic understanding capability of the visual encoder while training for the generation task, avoiding catastrophic forgetting; the pixel distillation loss enhances the encoder's extraction of details such as underlying textures and edges, further improving the precision of the reconstructed image; and the adversarial loss significantly improves the realism and texture details of the reconstructed image. This comprehensive training strategy enables the obtained target visual encoder to achieve an optimal balance between image generation and image understanding tasks, generating high-quality images while maintaining strong semantic understanding capabilities, thus providing a more robust and efficient foundational model for subsequent image processing tasks.

[0098] Based on the above description, see Figure 2a The schematic diagram illustrates the core architecture of the image processing method provided in this embodiment, as follows: The visual encoder to be trained is responsible for extracting high-level semantic features of the image, employing a slow update strategy to protect pre-trained weights; both the semantic teacher (visual encoder) and the pixel teacher (pixel encoder) are fully frozen models, providing supervision signals for semantic distillation and pixel detail distillation, respectively; the dual-stream feature processing module contains two parallel paths, a semantic stream and a pixel stream, processing high-level semantics and low-level details, respectively; the fusion module dynamically fuses the outputs of the two paths into a unified latent variable; and the pixel decoder decodes the latent variable into a reconstructed image. This achieves decoupled learning and efficient fusion of semantic features and pixel detail features by introducing a dual-teacher supervision mechanism (semantic teacher and pixel teacher) and a dual-stream feature processing architecture.

[0099] Furthermore, traditional generative models rely on external strong visual representation models to extract image features during training, injecting these features as conditional signals into the diffusion model to guide the generation process. However, this approach has two problems: first, it introduces a strong dependence on the external model, increasing system complexity and inference costs; second, the representation of the external model may not match the data distribution of the current generation task. To address these issues, this embodiment proposes using the target visual encoder trained as described above to train the diffusion generative model. The core of this approach lies in utilizing a semantic self-distillation mechanism to solve these problems. During the generation task training phase, the visual encoder trained using the aforementioned double-stream distillation method is used. Processing input images Extract semantic features Since the encoder has been aligned with the original large-scale pre-trained model through semantic distillation, it retains strong semantic understanding capabilities; at the same time, it has participated in image reconstruction training, establishing an implicit association with the pixel space. Therefore, its output features are naturally suitable as conditional guidance for generative models, without relying on external models. In this embodiment, after the step of training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder, the method further includes: A diffusion generation model is obtained by training a diffusion generation model based on the target visual encoder; wherein, the target diffusion generation model is used to perform local or global editing processing on the input image.

[0100] The target visual encoder is a visual feature extraction model obtained through a specific training process (e.g., trained by balancing image reconstruction and understanding tasks, as described in the aforementioned embodiments). This encoder has a unique advantage: it can extract comprehensive visual features from the input image that include both high-level semantic information (e.g., object categories, scene layout, etc., which are helpful for image understanding tasks) and low-level pixel details (e.g., texture, edges, colors, etc., which are crucial for image generation and reconstruction). This balanced feature representation enables the target visual encoder to provide high-quality conditional information in subsequent generation tasks. As one possible implementation, the encoder can be a Transformer-based model, such as Vision Transformer (ViT) or its variants, with an optimized internal structure capable of capturing visual information at different scales. Alternatively, the encoder can be a Convolutional Neural Network (CNN)-based model, such as ResNet or EfficientNet, which progressively extracts hierarchical features of the image through multiple layers of convolution and pooling operations, and is specifically trained to maintain feature richness and discriminativeness.

[0101] Correspondingly, the diffusion generative model is a Markov chain-based generative model that gradually adds noise to the data to disrupt its structure, then learns the inverse process—that is, gradually recovers the data from the noise—to achieve high-quality data generation. It can learn complex data distributions and generate new samples similar to the training data distribution, or edit existing samples. As one possible implementation, the diffusion generative model can adopt the architecture of the Denoising Diffusion Probabilistic Model (DDPM), which includes a U-Net-structured neural network as a noise predictor, progressively denoising by predicting and subtracting noise at each step. As another possible implementation, the diffusion generative model can also be a Latent Diffusion Model (LDM), which performs diffusion and denoising operations in a latent space rather than a pixel space, thus significantly reducing computational costs and improving generation efficiency.

[0102] Accordingly, training the diffusion generative model aims to enable it to learn how to generate or edit images by utilizing features provided by the target visual encoder as conditions. One possible implementation is to include inputting image features extracted by the target visual encoder as conditional inputs into the U-Net structure of the diffusion model, for example, through cross-attention mechanisms or adaptive normalization layers, guiding the diffusion model to generate images conforming to specific semantics and styles during denoising. Another possible implementation is to use a joint optimization strategy, where the diffusion model's loss function, in addition to the standard denoising loss, may introduce perceptual or adversarial losses to ensure the quality and realism of the generated images, while utilizing features provided by the target visual encoder as additional supervision signals.

[0103] Correspondingly, a target diffusion generation model is a diffusion model that, after training, can generate or edit images based on specific conditions (such as text descriptions and image features). It can perform diverse image generation and editing tasks, such as text-to-image generation, image-to-image conversion, and local region editing. As one possible implementation, the target diffusion generation model can be configured to accept text embeddings and image features as conditional inputs, thereby enabling text-guided image editing, such as modifying image content or style based on text instructions. As another possible implementation, the target diffusion generation model can also be designed to support image mask input, thereby enabling local image editing, such as repairing, replacing, or stylizing specific regions of an image.

[0104] Correspondingly, image editing refers to operations that modify the content, style, or structure of an image. Local editing targets specific areas of an image, while global editing affects the entire image. Through editing, various applications can be achieved, such as modifying image content, repairing, style transfer, and adjusting attributes. As one possible implementation, local editing can include image inpainting, which fills in missing or damaged areas of an image; or image replacement, which replaces specific areas of an image with new content. As another possible implementation, global editing can include image style transfer, which applies the style of one image to another; or image content transformation, such as converting a daytime scene to a nighttime scene, or changing the overall attributes of objects in an image.

[0105] Based on this, efficient and high-quality image editing is achieved by first obtaining an optimized visual encoder that balances image understanding and generation capabilities, and then training a diffusion generative model based on this encoder. Specifically, after training the target visual encoder, it provides robust visual features that take into account both high-level semantics and low-level pixel details. These features are used as key conditional information during the training process of the diffusion generative model. When learning to recover images from noise, the diffusion model uses these visual features to guide its generation process, ensuring that the generated image is semantically consistent with the original image and has high fidelity at the pixel level. This approach avoids the problems of traditional methods, such as completely freezing the visual encoder, which leads to loss of detail, or performing full fine-tuning, which leads to a decrease in understanding capabilities. By integrating the rich features provided by the target visual encoder into the conditional generation mechanism of the diffusion model, the scheme in this embodiment enables the diffusion model to more accurately understand editing instructions and generate high-quality local or global editing results. This tight integration allows the model to efficiently complete tasks from image understanding to complex image editing within a unified framework, without relying on external models, thereby simplifying the system architecture and improving overall performance.

[0106] In practice, the semantic features extracted by the target visual encoder can be used. This is injected as a conditional input into the diffusion transformer. Within the training framework of the diffusion model, a clean image is given... Noise is added through the forward diffusion process to obtain : ,in For noise dispatch coefficient, For time steps. Diffusion converter Learning under conditions Guided noise prediction: During the inference phase, from random noise The process begins by generating an image through an iterative denoising process.

[0107] For example, suppose we need to perform local editing on an image containing a person, such as changing the color of their clothing, or global editing, such as changing the overall style of the image to a cartoon style. First, the original image is encoded using the target visual encoder trained in the aforementioned implementation, extracting comprehensive visual features containing its semantic content and pixel details. These features are then conditionally input into the diffusion generation model to be trained. When training the diffusion generation model, paired image data can be used, such as the original image and its corresponding edited image, or the original image and text instructions describing the editing content. During denoising, the diffusion model (e.g., a U-Net architecture) utilizes the image features provided by the target visual encoder and editing instructions (such as text embeddings or masking information) to guide generation. For example, in the various layers of U-Net, visual and text features can be fused through a cross-attention mechanism, enabling the model to understand and follow editing instructions during generation. After sufficient training, the resulting target diffusion generation model can receive the image to be edited and the editing instructions. For example, when a user inputs an image and the text instruction "change the clothing color to blue," the target visual encoder extracts image features, and the text encoder extracts text features. These features are then used as conditional inputs into the target diffusion generation model. Through an iterative denoising process, the model generates an image where the character's clothing color has been changed to blue, while keeping other areas of the image unchanged. Similarly, for global editing, such as "convert the image style to cartoon style," the target visual encoder extracts image features and combines them with the text or style features of "cartoon style" as conditions. The target diffusion generation model applies this style transfer across the entire image, generating a new image with a cartoon style.

[0108] In summary, the above-described approach effectively addresses the challenge of efficiently applying a trained visual encoder to more advanced image editing tasks. This scheme avoids dependence on external representation models, thereby reducing system complexity and inference latency. Since the target visual encoder itself is optimized to provide features that balance global semantics and local details, the diffusion generation model trained on this basis can better maintain semantic consistency and visual realism in image editing, avoiding artifacts and detail loss. This method enables the target diffusion generation model to flexibly perform local or global image editing tasks, such as precise local content replacement, image inpainting, style transfer, or scene reconstruction, with higher quality and consistency in the editing results. In this way, this embodiment provides a unified and efficient framework, allowing visual encoders that perform well in image understanding tasks to be seamlessly extended to complex image generation and editing applications, achieving effective reuse and expansion of model capabilities.

[0109] Building upon this, efficiently fusing text and image features to generate high-quality edited images while avoiding loss of detail and decreased comprehension remains a challenge. To address this, in this embodiment, training a diffusion generation model based on the target visual encoder to obtain a target diffusion generation model includes: The process involves: acquiring an edit sample image and image editing information; encoding the image editing information using a text segmenter to obtain text features; encoding the edit sample image using the target visual encoder to obtain image semantic features; constructing a context sequence based on the text features and image semantic features; inputting the context sequence and preset noise features into a diffusion generation model for processing to obtain an edit prediction image; and optimizing the diffusion generation model based on the edit prediction image and the edit label image corresponding to the edit sample image to obtain a target diffusion generation model.

[0110] In this context, the editable sample image refers to the original image that requires editing. Its purpose is to provide visual context and basic content for the image editing task, upon which the model will modify it. This image can be in any digital image format, such as JPEG or PNG. Image editing information refers to the instructions or conditions that guide the image editing operation. Its purpose is to clarify the editing goal and scope, guiding the model to generate an edited result that matches the user's intent. This information can exist in various forms; for example, it could be a text prompt describing the content to be edited, such as "make the sky blue"; it could be a user-drawn sketch or mask indicating the editing area or the desired shape; or it could be other forms of visual or semantic input.

[0111] Correspondingly, a text segmenter is a tool that converts natural language text into a machine-understandable numerical representation. Its role is to transform unstructured text editing information into a semantic representation in a high-dimensional vector space, capturing lexical, grammatical, and semantic information within the text. For example, a pre-trained model based on the Transformer architecture, such as BERT or CLIP, can be used to map text information into fixed-dimensional feature vectors. Text features are numerical vectors obtained by encoding image editing information using a text segmenter. These features represent the semantic content of the text instructions and are the foundation for the model to understand the editing intent. The target visual encoder is a trained visual encoder whose role is to encode the input edited sample image into image semantic features with rich semantic information. This encoder is specially trained to preserve the high-level semantic understanding of the image while also taking into account low-level pixel details, thus avoiding the catastrophic forgetting or loss of detail problems in traditional methods. For example, this encoder can be an encoder based on a convolutional neural network or Transformer architecture, whose output can effectively capture the global structure and local texture of the image. Image semantic features are numerical vectors obtained by encoding the edit sample image by the target visual encoder. These features represent the visual semantic content of the image and are the visual basis for the model to perform image editing.

[0112] Correspondingly, the context sequence is a unified multimodal representation formed by fusing textual features and image semantic features. Its role is to provide comprehensive conditional information for the diffusion generation model, enabling the model to simultaneously understand textual instructions and image content, thereby generating edited results that meet both requirements. Construction methods can include, but are not limited to: simply concatenating the two features; or using attention mechanisms (such as cross-attention) to allow textual features to guide the transformation of image features, and vice versa; or using a dedicated fusion module to interact and integrate the two features. Preset noise features are random noise introduced during the training of the diffusion model, typically random vectors following a Gaussian distribution. Their role is to provide randomness to the generation process, enabling the model to generate diverse image results and helping it learn to progressively denoise from noise to generate clear images. The diffusion generation model is a generative model based on a diffusion process, generating images by progressively adding noise and denoising. Its role is to predict and generate images that conform to editing instructions given a context sequence and noise features. For example, a denoising network based on the U-Net architecture can be used as the core of the diffusion generation model, which can handle feature information at different scales. The edited prediction image is the image output by the diffusion generation model after processing the context sequence and preset noise features. This image is the prediction result after the model performs editing operations on the edited sample image.

[0113] Correspondingly, the edit label image is the ground truth image corresponding to the edit sample image and image editing information, i.e., the "ground truth" image. Its role is to serve as a supervisory signal for training the diffusion generation model, guiding the model to learn how to generate high-quality edit images. Optimization refers to the process of minimizing the difference between the output edit prediction image and the edit label image by adjusting the internal parameters of the diffusion generation model. Its purpose is to improve the model's generation ability and editing accuracy. The optimization process typically involves defining one or more loss functions (e.g., pixel-level L1 or L2 loss, perceptual loss, adversarial loss, etc.) to quantify the difference between the predicted image and the label image, and using optimizers (e.g., Adam, SGD, etc.) to update the model parameters.

[0114] Based on this, the first step is to acquire the edit sample image to be edited and the image editing information guiding the editing operation. To effectively utilize these multimodal inputs, the scheme inputs the image editing information into a text segmenter for encoding, thereby obtaining text features representing the text semantics. Simultaneously, the edit sample image is input into a target visual encoder for encoding, to obtain image semantic features capturing the visual semantics of the image. This target visual encoder is specially trained; it not only retains strong semantic understanding capabilities but also establishes implicit associations with the pixel space, effectively extracting global semantics and local details of the image. This avoids the detail loss problem caused by traditional frozen encoders and the decreased understanding capability problem caused by fully fine-tuned encoders. Subsequently, text features and image semantic features are used to construct a context sequence. This step is crucial for achieving deep fusion of multimodal information, organically combining the intent of the text instructions with the visual content of the image to form a unified and rich conditional representation. This context sequence, along with preset noise features, is fed into a diffusion generation model. Under these conditions, the diffusion generation model gradually recovers the edit prediction image from the noise through an iterative denoising process. The introduction of pre-defined noise features enables the model to learn and generate diverse editing results, enhancing its adaptability to complex editing tasks. Finally, to ensure the diffusion generation model can generate high-quality, expected edited images, the scheme optimizes the model based on the predicted edit image and the corresponding edit label image. By comparing the difference between the model's predictions and the true labels, and using a loss function for backpropagation, the model's parameters are continuously adjusted and optimized until it can stably generate high-fidelity, detailed, and semantically accurate edited images, thus obtaining the target diffusion generation model. Through the above process, this embodiment cleverly utilizes the advantages of a pre-trained target visual encoder, achieving efficient fusion of text and image features, avoiding dependence on external strong visual representation models, and reducing system complexity and inference latency. Simultaneously, since the target visual encoder has resolved the conflict between understanding ability and generation quality during training, its output image semantic features are naturally suitable as conditional guidance for the diffusion generation model, enabling the finally trained target diffusion generation model to simultaneously achieve high-quality generation and accurate semantic understanding in image editing tasks.

[0115] For example, suppose we need to train a diffusion-generative model that can edit images based on text instructions. First, prepare a dataset containing 'edit sample images', corresponding 'image editing information' (e.g., text descriptions), and 'edit label images'. Specifically, when given an 'edit sample image' (e.g., a landscape photo) and 'image editing information' (e.g., the text "turn the sky to the orange-red of sunset"), we first use a pre-trained 'text segmenter', such as the CLIP text encoder based on the Transformer architecture, to encode the text instruction "turn the sky to the orange-red of sunset," thus obtaining a 1x768-dimensional 'text feature' vector. Simultaneously, the landscape photo is input into a 'target visual encoder' that has been semantically distilled. This encoder can be a ResNet-50 or VisionTransformer model, and its output is a 1x1024-dimensional 'image semantic feature' vector, which contains both the global semantic information of the image and preserves important local details. Next, to construct the `context sequence`, the obtained `text feature` vector and `image semantic feature` vector can be input into a multimodal fusion module. This fusion module can employ a cross-attention mechanism, where the `text features` serve as the query and the `image semantic features` as the key and value. Attention calculation guides and fuses the text with the image features, ultimately outputting a `context sequence` that integrates text and image information. Simultaneously, a `preset noise feature` matching the size of the `edit sample image` is generated, such as a random noise map following a standard Gaussian distribution. Then, the constructed `context sequence` and `preset noise feature` are input into a `diffusion generation model`. This `diffusion generation model` can be a denoising network based on the U-Net architecture, whose input is a noisy image and time step information, and whose output is the predicted noise. During training, the model, guided by the `context sequence`, gradually denoises from the `preset noise feature` to generate an `edit prediction image`. Finally, to optimize the `diffusion generation model`, the generated `edit prediction image` is compared with the corresponding `edit label image` (e.g., a real sunset sky landscape photo). Various loss functions can be used for optimization. For example, pixel-level L1 loss is used to measure the similarity of image content, perceptual loss is used to measure the similarity of high-level features, and adversarial loss is used to improve the realism of the generated images. Using algorithms such as the Adam optimizer, the parameters of the 'diffusion generation model' are iteratively updated based on these loss functions until the model can stably generate high-quality, text-instruction-compliant, and richly detailed edited images, thus obtaining the 'target diffusion generation model'.

[0116] In summary, through sophisticated feature extraction, fusion, and optimization mechanisms, the trained target diffusion generation model can efficiently perform image editing. It achieves high-quality image generation without relying on external strong visual representation models, while maintaining the original image understanding capabilities of the visual encoder, significantly improving the performance and efficiency of image editing tasks. Furthermore, the fusion method may not be deep or structured enough, resulting in insufficient multimodal information interaction and an inability to fully capture the semantic relationships between text and images. This affects the input quality of subsequent diffusion generation models, thereby reducing the accuracy of image editing and the generation effect. To address this, in this embodiment, constructing a context sequence based on the text features and the image semantic features includes: Determine the query embedding features; input the query embedding features, the text features, and the image semantic features into the target language model for processing to obtain multimodal hidden state features and target query embedding features; use a connector to process the multimodal hidden state features and the target query embedding features to obtain a context sequence.

[0117] Query embeddings serve as the initial vector representation used to guide the fusion of multimodal information. Their role is to provide a clear starting point or reference point, guiding subsequent language models to purposefully interact and align text and image features. One possible implementation is that the query embedding can be a pre-defined, learnable vector that is optimized during model training. Another possible implementation is that the query embedding can be generated from task-related inputs (e.g., a specific token or a simplified task description) through a small neural network layer, tailored to specific task requirements. Inputting query embeddings, text features, and image semantic features into the target language model aims to achieve deep interaction and fusion of multimodal information, leveraging the powerful capabilities of language models (such as attention mechanisms) to jointly encode and semantically align features from different modalities. As another possible implementation, the target language model can be a large-scale language model (LLM) based on the Transformer architecture, such as the GPT series, BERT series, or their variants, pre-trained or fine-tuned with multimodal data. As another possible implementation, the target language model can also be a specially designed multimodal Transformer model whose input layer can simultaneously receive and process embedding features from different modalities, and achieve feature fusion through a cross-modal attention mechanism. Through the above processing, multimodal hidden state features and target query embedding features can be obtained. The multimodal hidden state features are intermediate representations obtained after language model processing, fusing text and image information, and containing rich semantic associations; the target query embedding features are query embeddings processed and updated by the language model, absorbing contextual information from text and images, becoming more instructive. These features are usually direct outputs of the target language model, such as the output of the hidden layers within the model. Finally, a connector is used to process the multimodal hidden state features and target query embedding features, aiming to further integrate and optimize the features output by the language model, forming a unified and structured context sequence that adapts to the input requirements of subsequent diffusion generation models. As one possible implementation, the connector can be a simple linear layer (fully connected layer) that concatenates two feature vectors and then performs a dimensionality transformation. As another possible implementation, the connector could also be a multilayer perceptron (MLP) or a lightweight Transformer encoder for more complex feature fusion and sequence rearrangement.

[0118] Based on this, by introducing query embedding features and a target language model, deep interaction and fusion of text features and image semantic features are achieved, thereby constructing a higher-quality context sequence. Specifically, firstly, query embedding features are determined, providing an initial reference point for the multimodal fusion process and guiding feature interaction with a clear starting point, avoiding semantic confusion caused by a lack of guidance during direct fusion. Subsequently, query embedding features, text features, and image semantic features are input into the target language model for processing. The model's powerful capabilities (such as attention mechanisms) are utilized to jointly encode the multi-source inputs, generating multimodal hidden state features and target query embedding features. This solves the problem of insufficient interaction in simple fusion methods, enabling text and image features to be fully aligned and enriched at the semantic level. After obtaining the multimodal hidden state features and target query embedding features, these features not only retain the integrity of the original information but also enhance their representational capabilities through updates to the target language model, providing a better intermediate state for subsequent steps. Finally, a connector is used to process the multimodal hidden state features and target query embedding features, further integrating and optimizing these features to form a unified and structured context sequence. This contextual sequence, as input to the diffusion generative model, can more accurately guide the image generation process, especially in image editing tasks. It can more precisely capture the complex semantic relationships between text descriptions and image content, thereby improving editing accuracy and the quality of generated images. This approach, combined with the aforementioned method of training a diffusion generative model based on a target visual encoder, allows the image semantic features extracted by the visual encoder trained through double-stream distillation to be efficiently combined with text features through this deep fusion mechanism. This provides high-quality conditional information for the diffusion generative model, enabling robust and efficient image editing without relying on external representation models.

[0119] For example, when determining the query embedding feature, it can be a vector of length N, such as a vector generated by an embedding layer from a specific task token (e.g., `[EDIT_QUERY]`). When inputting the query embedding feature, text features, and image semantic features into the target language model for processing, the target language model can be a pre-trained Transformer model with multimodal capabilities, such as a model based on the Flamingo or BLIP-2 architecture. Upon input, the query embedding feature, text features (e.g., a feature sequence generated by the CLIP text encoder), and image semantic features (e.g., a feature sequence obtained after encoding an edited sample image by the target visual encoder) can be concatenated in a specific order, with an appropriate attention mask added, and then input into the Transformer's encoder layer. After obtaining the multimodal hidden state features and the target query embedding feature, the hidden states corresponding to the text and image input positions can be extracted from the output of the last layer of the target language model as multimodal hidden state features, and the outputs corresponding to the query embedding feature positions can be extracted as the target query embedding feature. Finally, when processing the multimodal hidden state features and target query embedding features using the connector, the connector can be an MLP consisting of two linear layers with a ReLU activation function in between. It receives the concatenated multimodal hidden state features and target query embedding features and maps them to a fixed-length vector as the final context sequence.

[0120] In summary, the above processing effectively addresses the problem of insufficient multimodal information interaction, significantly improving the fusion depth and structuring level of text features and image semantic features. This allows the constructed context sequence to more fully capture the semantic relationships between text and images, thus providing higher-quality input for subsequent diffusion generation models. Ultimately, this scheme significantly improves the accuracy and generation effect of image editing, resulting in stronger semantic consistency between the edited predicted image and the edited labeled image, and a marked improvement in the detail and overall quality of the generated image.

[0121] Based on the above description, see Figure 2bThe diagram illustrates the training of the diffusion generation model in the image processing method provided in this embodiment. For the input image and text instructions, the text instructions can be encoded by a text segmenter, and the input image's semantic features are extracted by a target visual encoder trained through self-distillation. Both, along with the query embedding, are input into a large language model. The large language model outputs multimodal hidden states and query embeddings, which, after being processed by a connector, are input into a diffusion transformer along with noise to generate the target image. The core of semantic self-distillation lies in the fact that the semantic features extracted by the visual encoder after double-stream distillation training can be directly used to guide the generation process without relying on an external representation model. In other words, the main mechanism for training the diffusion generation model in this embodiment is semantic self-distillation. Its core meaning is that the generation model directly uses the features provided by its own understanding module (visual encoder) to guide generation, forming a closed loop of "understanding-generation," rather than relying on an external model. This brings two enhancement effects: firstly, understanding guides generation; the semantic features provided by the visual encoder contain a deep understanding of the image content, helping the generation model to more accurately grasp the semantic structure of the generated target. Since generation and understanding share the same semantic space, the generated image is semantically more consistent with the model's own understanding logic, improving the semantic consistency of the generation. On the other hand, generation feeds back into understanding: when the model is trained on data from a specific vertical domain (such as e-commerce or medical imaging), the visual encoder further learns the visual patterns and semantic concepts unique to that domain, thereby improving its understanding of that domain. This enhanced ability further improves the quality of generation, forming a positive cycle.

[0122] Therefore, the image processing method provided in this embodiment mainly consists of two stages in training the visual encoder and the diffusion generation model. Stage one is two-stream distillation reconstruction training. A joint loss function is used to train the entire reconstruction framework, including the trainable visual encoder, two-stream feature processing module, fusion module, and pixel decoder. During training, the semantic teacher and pixel teacher are frozen to provide stable distillation supervision for the student network. The visual encoder is used for protective fine-tuning with a low learning rate. Adversarial loss is enabled in the later stages of training to further improve reconstruction quality. Stage two is semantic distillation generation training. The visual encoder obtained in Stage one is fixed (or fine-tuned further with a low learning rate), and the diffusion transformer is trained. The semantic features extracted by the visual encoder are used as conditions to guide the diffusion model to learn image generation.

[0123] See Figure 3 , Figure 3 A flowchart of a second image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0124] Step S302: Determine the visual features and pixel features corresponding to the original sample image, and process the original sample image using the visual encoder to be trained to obtain basic features.

[0125] Step S304: Construct visual semantic features based on the visual features and the basic features, and construct pixel semantic features based on the pixel features and the basic features.

[0126] Step S306: The visual semantic features and the pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image.

[0127] Step S308: Train the visual encoder to be trained based on the original sample image and the reconstructed image, and optimize the diffusion generation model based on the trained visual encoder to obtain the target diffusion generation model.

[0128] The second image processing method provided in this embodiment, for any content not described in detail, can be referred to the same or corresponding descriptions in the above embodiments, and will not be elaborated further in this embodiment.

[0129] In practical applications, existing multimodal large language models have made significant progress in image understanding tasks, but achieving high-quality image generation and understanding simultaneously in a unified model still faces major challenges. Specifically, when the visual encoder is completely frozen, the reconstructed image suffers from obvious artifacts and blurring because the pre-trained visual encoder mainly optimizes high-level semantic information and lacks sufficient representation of low-level pixel-level details (such as texture and edges). On the other hand, while full fine-tuning of the visual encoder can improve the generation quality, it destroys its original general visual semantic representation, leading to catastrophic forgetting problems and significantly degrading the model's performance in understanding tasks such as classification and detection.

[0130] To address this, the second image processing method provided in this embodiment first uses the original sample image to extract visual and pixel features. Visual features are obtained through the intermediate layer output of a network (such as ResNet) pre-trained on a large-scale image classification dataset. These features focus on capturing global semantic information of the image, such as identifying the main object categories in the scene. Pixel features are directly extracted from the original sample image through convolutional layer stacking or edge detection algorithms, focusing on low-level detail information such as texture structure and edge sharpness. Simultaneously, the original sample image is input to a visual encoder to be trained. This encoder, as a trainable convolutional neural network component, outputs basic features, serving as a common foundational representation for subsequent feature construction.

[0131] Furthermore, visual semantic features are constructed by fusing visual features with basic features, such as through feature concatenation or multilayer perceptron mapping, ensuring that high-level semantic information is not covered during training, thereby maintaining the model's discriminative ability in understanding tasks. Pixel semantic features are constructed by fusing pixel features with basic features, effectively preserving low-level detail information and providing necessary texture and edge guidance for image reconstruction. Subsequently, visual semantic features and pixel semantic features are fused, for example through channel-level concatenation or element-wise summation, to form a comprehensive feature containing both global semantics and local details. This fused feature is decoded by a pixel decoder, which progressively converts the features into a reconstructed image with the same size as the original sample image through deconvolution layers or upsampling modules.

[0132] Finally, a reconstruction loss function (such as mean squared error) is calculated based on the original sample images and the reconstructed images to optimize the parameters of the visual encoder to be trained. Through backpropagation, the encoder refines its internal feature representation while minimizing pixel-level differences, avoiding the catastrophic forgetting problem. After training, the parameters of the diffusion generation model are further adjusted based on the optimized visual encoder to obtain the target diffusion generation model, which can efficiently support generation tasks such as image editing.

[0133] In summary, the core of this embodiment lies in combining visual features with basic features to form visual semantic features to maintain understanding capabilities through a dual-stream feature construction mechanism, while simultaneously combining pixel features with basic features to form pixel semantic features to improve reconstruction quality. This design avoids catastrophic forgetting caused by full fine-tuning and overcomes the detail loss problem caused by frozen encoders. Through the above technical solutions, synergistic optimization of image understanding and generation tasks is achieved, significantly improving the robustness and applicability of the model in multimodal tasks. For example, when processing landscape images with complex backgrounds, this method can simultaneously retain the recognition ability of high-level semantics such as "mountains" and "rivers" and generate high-quality reconstruction results with clear rock textures and water ripple details, effectively solving the core contradictions in existing technologies.

[0134] See Figure 4 , Figure 4 A flowchart of a third image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0135] Step S402: Obtain the image to be processed and identify the image region to be edited in the image to be processed.

[0136] Step S404: The target diffusion generation model is used to edit the image region to be edited in the image to be processed to obtain the target image corresponding to the image to be processed. The target diffusion generation model is determined by the above method.

[0137] The third image processing method provided in this embodiment, for any content not described in detail, can be referred to the same or corresponding descriptions in the above embodiments, and will not be elaborated further in this embodiment.

[0138] Specifically, acquiring the image to be processed refers to acquiring the original image data for which image editing operations are required. This image data can be a digital image file acquired through an image acquisition device (e.g., a digital camera, scanner), image data read from a storage medium (e.g., a hard disk drive, solid-state drive, cloud storage service), or a real-time image stream received via a network. This step provides the basic input for subsequent image recognition and editing operations. Identifying the editable image region in the image to be processed refers to locating and marking specific image regions that need to be modified or removed within the image to be processed, based on preset rules, standards, or models. This identification process can utilize image recognition algorithms, such as object detection models (e.g., YOLO, Faster R-CNN) or semantic segmentation models (e.g., U-Net, DeepLab), to automatically detect specific content in the image and output its location information (e.g., bounding boxes or pixel masks); alternatively, the editable image region can be specified through manual annotation, user interactive selection, etc. This step aims to accurately locate the target to be edited, avoiding accidental operations on other parts of the image, thereby improving the targeting and efficiency of the editing process. The editing of the image region to be edited in the image to be processed using the target diffusion generation model refers to using a specifically trained and optimized diffusion generation model to perform operations such as content replacement, repair, or style transfer on the identified image region to be edited. This model can receive the image to be processed and the mask of the image region to be edited as input, and generate new pixel content to cover or replace the region, or generate within a specified region while maintaining the integrity of other parts of the image. Obtaining the target image corresponding to the image to be processed refers to the final image after editing, in which the image region to be edited has been corrected or replaced, and the overall content meets the expected requirements. The target image can be a complete image directly output by the diffusion generation model, or it can be an image formed by splicing and fusing the local editing results generated by the model with the non-edited area of ​​the original image. The target diffusion generation model is determined through the above method, emphasizing that the diffusion generation model used is not a general model, but rather obtained through a specific training process (e.g., visual encoder training and diffusion generation model optimization). This model operates within a unified framework. By training a visual encoder to effectively fuse visual and pixel features, and then optimizing the diffusion generation model based on this, a model that performs exceptionally well in both generation quality and understanding capabilities is obtained.

[0139] In practice, when editing specific regions of an image (such as the region to be edited), it may be difficult to achieve precise modifications efficiently, or additional complex processing may be required, leading to inefficient editing processes, unstable quality, and unavoidable dependence on external models. To address this, the third image processing method provided in this embodiment is applied to content understanding scenarios. It acquires the image to be processed as input and accurately identifies the region to be edited, providing a clear target for subsequent editing operations. Then, a specially trained and optimized target diffusion generation model is used to edit these regions. This target diffusion generation model is determined based on the aforementioned method, meaning that it fully considers the fusion of visual and pixel features during training and is optimized through multiple loss functions (including reconstruction loss, perceptual loss, semantic distillation loss, pixel distillation loss, and adversarial loss). This training method enables the model to simultaneously possess the ability to understand high-level semantics of the image and generate low-level pixel details, thus generating high-quality content that is semantically consistent with the overall image content and seamlessly connected at the pixel level when editing the region to be edited. In this way, the solution in this embodiment avoids the artifacts and blurring caused by freezing the visual encoder in traditional methods, as well as the catastrophic forgetting problem caused by full fine-tuning of the visual encoder, ensuring the quality of the editing results and the model's understanding ability. Ultimately, the obtained target image not only repairs the area to be edited but also maintains the overall consistency and high fidelity of the image. For example, in a content understanding and intelligent repair scenario, the system first acquires the image to be processed uploaded by the user. Next, it analyzes the image using a pre-trained object detection model or semantic segmentation model to identify and locate the region to be edited, such as sensitive symbols, inappropriate text, or prohibited items. The identification result can be a mask image that precisely marks the pixel positions of the region to be edited. Then, the image to be processed and the identified mask of the region to be edited are input into a target diffusion generation model. This target diffusion generation model, determined through the above method, possesses powerful image understanding and generation capabilities. Based on the contextual information of the image to be processed, the model generates new pixel content within the region to be edited, conforming to platform requirements, to replace or repair the original content. For example, if the region to be edited is an inappropriate symbol, the model can generate a blank area or a replacement symbol that conforms to platform requirements, ensuring it blends naturally with the surrounding image content without obvious boundaries or artifacts. Finally, the image output by the model is the target image, where the region to be edited has been successfully edited, and the overall image content meets the mask requirements.

[0140] In summary, the above processing method enables efficient and precise editing of specific regions in an image (such as the region to be edited), solving the problems of low efficiency, unstable quality, and dependence on external models in traditional methods for local editing. This method utilizes an optimized, trained target diffusion generation model, ensuring both the overall semantic consistency of the image and the generation of high-quality pixel details during the editing process. This significantly improves the automation level of image editing and the visual effect of the edited results, making it particularly suitable for scenarios requiring intelligent local image restoration, such as content understanding.

[0141] See Figure 5 , Figure 5 A flowchart of a fourth image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0142] Step S502: Receive the image to be edited and determine the editing information corresponding to the image to be edited.

[0143] Step S504: The target diffusion generation model is used to edit the image to be edited according to the editing information to obtain the target image corresponding to the image to be edited, wherein the target diffusion generation model is determined by the above method.

[0144] The fourth image processing method provided in this embodiment, for any content not described in detail, can be referred to the same or corresponding descriptions in the above embodiments, and will not be elaborated further in this embodiment.

[0145] Specifically, receiving the image to be edited refers to the system acquiring the original image provided by the user or an external source as the basis for subsequent editing operations. This image can be an image file uploaded through the user interface, such as JPEG or PNG format images, or image data retrieved from image databases or cloud storage services, or an image stream acquired by a real-time capture device (such as a camera). Determining the editing information corresponding to the image to be edited refers to the system recognizing and parsing the image modification instructions specified by the user or preset rules. This editing information can take various forms, such as natural language text instructions, like "change the sky to blue" or "remove people from the background"; it can also be selections or annotations drawn by the user on the image using graphical interface tools (such as brushes or selection boxes) to specify the local area to be edited; or it can be preset editing templates or style parameters, such as "apply oil painting style" or "adjust to black and white photo". Using a target diffusion generation model to edit the image to be edited according to the editing information means taking the image to be edited and the editing information as input, and modifying the image content through a specifically trained diffusion generation model. This model can make local or global adjustments to the image based on the editing information, while maintaining the overall quality and consistency of the image. For example, the model can receive embedded representations of editing information (such as text embeddings or mask information) as conditional input to guide the diffusion process and generate images that meet the conditions; or, during iterative denoising, the image can be gradually adjusted in conjunction with the editing information. Obtaining the target image corresponding to the image to be edited refers to the final image output by the target diffusion generation model after the above editing processing. This target image is the result that meets the user's editing needs and can be saved as an image file or displayed in real-time on the user interface for preview and further operation. The target diffusion generation model is determined through a training method, meaning that the model is not an untrained general model, but rather has undergone a specialized training process, such as training using the method described in claim 10, to enable it to generate and understand high-quality images, thereby effectively handling complex image editing tasks and avoiding catastrophic forgetting or artifact problems that may occur in traditional methods.

[0146] In practice, efficiently and accurately editing images based on editing information while maintaining image quality is a challenge. To address this, this embodiment proposes a fourth image processing method. This method receives the image to be edited as input and combines it with explicit editing information, then utilizes an optimized target diffusion generation model for processing. The training process of this target diffusion generation model ensures a balance between image generation and image understanding tasks, enabling it to accurately understand the semantics of editing instructions and execute editing operations with high quality. During processing, the editing information serves as a conditional input, guiding the diffusion generation model to precisely adjust image features in the latent space. This allows for the modification or generation of specified content without destroying the original image structure and details. This mechanism enables the model to perform fine-grained local or global editing of the image based on user intent, ultimately generating a target image that meets the requirements. This method cleverly integrates editing instructions into the generation process of the diffusion model, allowing the model to perform purposeful image reconstruction based on semantic information, rather than simple pixel manipulation. This achieves efficient and accurate image editing while maintaining image quality.

[0147] For example, in a content creation assistance scenario, a user uploads a landscape photo as the image to be edited. After receiving the landscape photo, the user can input text instructions as editing information, such as "change the sky in the photo to the orange-red of a sunset and add a flowering tree to the grass in the foreground." The system inputs the landscape photo and text instructions into the target diffusion generation model. The model first parses the text instructions, understanding the semantic information of "orange-red sky at sunset" and "flowering tree," and combines this with the visual features of the landscape photo. Subsequently, during the diffusion process, the model conditionally generates and modifies the corresponding areas of the image based on this editing information. Specifically, the model might generate an orange-red gradient effect in the sky area and synthesize a flowering tree in the grass area that harmonizes with the surrounding environment. Finally, the model outputs an edited landscape photo where the sky is a sunset orange-red, a flowering tree has been added to the grass, and the overall image style is harmonious and unified, serving as the target image.

[0148] In summary, the above processing methods enable efficient and accurate image editing based on user-provided editing information. By employing a specifically trained target diffusion generation model, this model generates high-quality images while maintaining a strong understanding of the image content. This effectively avoids common problems in traditional image editing methods, such as artifacts, blurring, or loss of detail, and also prevents a decline in understanding capabilities due to model fine-tuning. This makes the image editing process more precise, generates higher-quality target images, and greatly enhances the flexibility and user experience of image editing, making it particularly suitable for scenarios requiring refined content creation and modification. See Figure 6 , Figure 6 A flowchart of a fifth image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0149] Step S602: Receive the image to be enhanced and determine the sample generation information corresponding to the image to be enhanced.

[0150] Step S604: The target diffusion generation model is used to edit the image to be enhanced according to the sample generation information to obtain the sample image corresponding to the image to be enhanced. The target diffusion generation model is determined by the above method.

[0151] The fifth image processing method provided in this embodiment, for any content not described in detail, can be referred to the same or corresponding descriptions in the above embodiments, and will not be elaborated further in this embodiment.

[0152] Specifically, receiving the image to be enhanced refers to the system acquiring the original image data that needs to be enhanced. This process can be achieved in various ways. For example, users can upload image files from local storage devices or select images from network paths via an interface; they can also acquire image data streams in real time through image acquisition devices such as cameras or scanners; in addition, the image to be enhanced can be read from a preset image database or cloud storage service.

[0153] The sample generation information corresponding to the image to be enhanced refers to the instructions or data set to guide the image enhancement process, describing the desired features or requirements of the generated sample image. This information can be provided actively by the user, for example, by entering a natural language description through a text input box, such as "generate a picture of a cat with more detail" or "replace the background with a forest"; it can also be provided by selecting preset enhancement templates or parameters, such as "sharpen," "increase contrast," or "style transfer to oil painting style"; or by providing a reference image or image region, indicating the desired style, texture, or content. In addition, the system can also automatically generate or recommend sample generation information based on the metadata of the image to be enhanced (such as shooting time, location, and device) or content analysis (such as identifying objects and scenes in the image).

[0154] Editing the image to be enhanced using a target diffusion generation model based on the sample generation information is the core step in image enhancement. This model is a specially trained diffusion model that modifies the image according to given guidance information to generate a qualified sample image. For example, the target diffusion generation model can be a Denoising Diffusion Probabilistic Model (DDPM) based on the U-Net architecture. Its conditional inputs include the encoded features of the image to be enhanced and sample generation information (such as text embeddings). The model iteratively transforms the noisy image into a target sample image through a denoising process. Alternatively, the target diffusion generation model can also be a Latent Diffusion Model (LDM), which performs diffusion and denoising operations in the latent space, thereby improving processing efficiency. The sample generation information serves as a conditional input, guiding the generation process of the latent space. Furthermore, the target diffusion generation model can also be a diffusion model based on the Transformer architecture, which can better handle long-range dependencies and multimodal information fusion, thus more accurately understanding the sample generation information and guiding image editing.

[0155] Obtaining the sample image corresponding to the image to be enhanced refers to the final image output that has been edited and processed to meet the requirements of sample generation information and has an enhancement effect. This sample image can be pixel-level image data directly output by the model and can be saved in common image formats such as PNG or JPEG. In some cases, the image data output by the model can be further optimized through post-processing modules, such as color correction and resolution enhancement, to obtain higher-quality sample images.

[0156] In practical implementation, for specific requirements of the image to be enhanced, such as generating sample images, there may be problems such as low quality of the generated images, loss of details, or inability to accurately meet the sample generation information. Specifically, existing methods are prone to blurred reconstruction and artifacts due to visual encoder freezing in image enhancement tasks, or a decline in understanding ability due to fine-tuning, thus affecting the realism and applicability of the sample images. To address this, this embodiment proposes a fifth image processing method. By receiving the image to be enhanced and determining its corresponding sample generation information, it provides clear input and guidance for subsequent image editing. The core lies in utilizing a specially trained target diffusion generation model. This model fuses visual semantic features and pixel semantic features, and balances reconstruction loss, perceptual loss, semantic distillation loss, pixel distillation loss, and adversarial loss during training, thereby achieving characteristics that possess both strong global semantic understanding capabilities and the ability to preserve and generate rich pixel-level details. When the model receives the image to be enhanced and the sample generation information, it can accurately understand the user's intent and perform targeted editing of the image based on this information. For example, in data augmentation scenarios, the model can modify the original image locally or globally based on the instruction to "generate challenging samples with specific semantic features," generating sample images that are more challenging in terms of lighting, texture, and background complexity. This approach avoids the reconstruction blurring and artifacts caused by visual encoder freezing in traditional methods, and also avoids the catastrophic forgetting problem caused by full fine-tuning of the visual encoder, thus ensuring the high quality and realism of the generated sample images. In this way, the solution in this embodiment can effectively transform the image to be augmented into a sample image that meets specific requirements, greatly improving the flexibility and effectiveness of image augmentation tasks.

[0157] For example, suppose we need to augment the training dataset of an image classification model to improve its generalization ability to complex scenes. First, the system receives a batch of original training images as images to be augmented, such as animal pictures taken in standard environments. Next, for these images, sample generation information is determined. For example, for a picture of a cat, the sample generation information could be a text description, such as "generate a picture of a cat in dim light with a cluttered background," or a pre-defined combination of augmentation parameters indicating the addition of noise and blur. Then, using a pre-trained target diffusion generation model, the images to be augmented and the corresponding sample generation information are input into the model for editing. The model modifies the original cat image according to the text description or parameter instructions, such as adjusting lighting conditions, adding background elements, or introducing specific types of noise. Finally, the model outputs the edited sample image, i.e., a picture of a cat in dim light with a cluttered background. These generated sample images can be added to the original training dataset to train the classification model, thereby improving the model's recognition accuracy and robustness in complex real-world scenes.

[0158] In summary, the above processing effectively addresses the problems of low-quality generated images, loss of detail, or inaccurate fulfillment of sample generation information in existing image enhancement methods. This scheme utilizes an optimized and trained target diffusion generation model to ensure accurate understanding of high-level semantics while generating rich and realistic pixel-level details during image enhancement, thus avoiding artifacts, blurring, and loss of detail common in traditional methods. Furthermore, guided by precise sample generation information, this scheme enables fine-grained control over the image editing process, ensuring that the generated sample images accurately meet specific enhancement requirements. This significantly improves the realism and usability of the generated samples, making it particularly suitable for applications such as data augmentation and difficult sample generation.

[0159] The following is in conjunction with the appendix Figure 7 Taking the application of the image processing method provided in this specification in content understanding and intelligent repair scenarios as an example, the image processing method will be further explained. Figure 7 A timing flowchart of a first image processing method provided in one embodiment of this specification is shown.

[0160] Image processing methods are applied to content application platforms, which include client and server sides. The client is the user's terminal device; users can publish content through the client. The server specifically refers to the server that provides content publishing services to users and supports content understanding and repair of the image and text content to be published. For example... Figure 7 As shown, the specific implementation method in the content understanding and intelligent repair scenario is as follows: On the server side, the visual and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain the basic features.

[0161] On the server side, visual semantic features are constructed based on visual features and basic features, and pixel semantic features are constructed based on pixel features and basic features.

[0162] On the server side, visual semantic features and pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image.

[0163] On the server side, the visual encoder to be trained is trained based on the original sample images and the reconstructed images, and the diffusion generation model is optimized based on the trained visual encoder to obtain the target diffusion generation model.

[0164] On the server side, the target diffusion generation model is deployed in content understanding and intelligent repair scenarios.

[0165] The client uploads the image to be processed on the content editing page and sends the image to the server.

[0166] On the server side, the image region to be edited in the image to be processed is identified, and the target diffusion generation model is used to edit the image region to be edited in the image to be processed to obtain the target image corresponding to the image to be processed.

[0167] The server sends the target image to the client.

[0168] The client submits the published content associated with the target image on the content editing page, and then submits the target image and the published content to the server.

[0169] On the server side, the target content is published on the target account based on the target image and the published content.

[0170] In summary, the above processing method enables efficient and precise editing of specific regions in an image (such as the region to be edited), solving the problems of low efficiency, unstable quality, and dependence on external models in traditional methods for local editing. This method utilizes an optimized, trained target diffusion generation model, ensuring both the overall semantic consistency of the image and the generation of high-quality pixel details during the editing process. This significantly improves the automation level of image editing and the visual effect of the edited results, making it particularly suitable for scenarios requiring intelligent local image restoration, such as content understanding.

[0171] The following is in conjunction with the appendix Figure 8Taking the application of the image processing method provided in this specification in a content creation assistance scenario as an example, the image processing method will be further explained. Among them, Figure 8 A timing flowchart of a second image processing method provided in one embodiment of this specification is shown.

[0172] Image processing methods are applied to content application platforms, which include client and server sides. The client is the user's terminal device; users can publish content through the client. The server specifically refers to the server that provides content publishing services to users and supports assisted image creation based on user-submitted images. For example... Figure 8 As shown, the specific implementation method in the content creation assistance scenario is as follows: On the server side, the visual and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain the basic features.

[0173] On the server side, visual semantic features are constructed based on visual features and basic features, and pixel semantic features are constructed based on pixel features and basic features; On the server side, visual semantic features and pixel semantic features are fused together, and the pixel decoder is used to decode the fusion result to obtain the reconstructed image corresponding to the original sample image. On the server side, the visual encoder to be trained is trained based on the original sample images and the reconstructed images, and the diffusion generation model is optimized based on the trained visual encoder to obtain the target diffusion generation model.

[0174] On the server side, the target diffusion generation model is deployed in content creation assistance scenarios.

[0175] The client uploads the image to be edited on the image editing page and sends the image to the server.

[0176] On the server side, the editing information corresponding to the image to be edited is determined, and the target diffusion generation model is used to edit the image to be edited according to the editing information to obtain the target image corresponding to the image to be edited.

[0177] The server sends the target image to the client.

[0178] The client displays the target image on the image editing page.

[0179] In summary, the above processing methods enable efficient and accurate image editing based on user-provided editing information. By employing a specifically trained target diffusion generation model, this model generates high-quality images while maintaining a strong understanding of the image content. This effectively avoids common problems in traditional image editing methods, such as artifacts, blurring, or loss of detail, and also prevents a decline in understanding capabilities due to model fine-tuning. This makes the image editing process more precise, generates higher-quality target images, and greatly enhances the flexibility and user experience of image editing, making it particularly suitable for scenarios requiring refined content creation and modification. The following is in conjunction with the appendix Figure 9 Taking the application of the image processing method provided in this specification in the scenario of enhancing sample construction as an example, the image processing method will be further explained. Among them, Figure 9 A timing flowchart of a third image processing method provided in one embodiment of this specification is shown.

[0180] Image processing methods are applied on the server side, which communicates with the client. The client is the user's terminal device; the user can construct augmented samples through the client. The server specifically refers to the service that provides augmented sample construction services to the user. For example... Figure 9 As shown, the specific implementation method in the augmented sample construction scenario is as follows: On the server side, the visual and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain the basic features.

[0181] On the server side, visual semantic features are constructed based on visual features and basic features, and pixel semantic features are constructed based on pixel features and basic features; On the server side, visual semantic features and pixel semantic features are fused together, and the pixel decoder is used to decode the fusion result to obtain the reconstructed image corresponding to the original sample image. On the server side, the visual encoder to be trained is trained based on the original sample images and the reconstructed images, and the diffusion generation model is optimized based on the trained visual encoder to obtain the target diffusion generation model.

[0182] On the server side, the target diffusion generation model is deployed in the enhanced sample construction scenario.

[0183] The client uploads the image to be enhanced on the sample construction page and sends the image to the server.

[0184] On the server side, the sample generation information corresponding to the image to be enhanced is determined, and the target diffusion generation model is used to edit and process the image to be enhanced according to the sample generation information to obtain the sample image corresponding to the image to be enhanced.

[0185] The server sends the target image to the client.

[0186] In summary, the above processing effectively addresses the problems of low-quality generated images, loss of detail, or inaccurate fulfillment of sample generation information in existing image enhancement methods. This scheme utilizes an optimized and trained target diffusion generation model to ensure accurate understanding of high-level semantics while generating rich and realistic pixel-level details during image enhancement, thus avoiding artifacts, blurring, and loss of detail common in traditional methods. Furthermore, guided by precise sample generation information, this scheme enables fine-grained control over the image editing process, ensuring that the generated sample images accurately meet specific enhancement requirements. This significantly improves the realism and usability of the generated samples, making it particularly suitable for applications such as data augmentation and difficult sample generation.

[0187] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 10 A schematic diagram of the structure of a first image processing apparatus provided in one embodiment of this specification is shown. Figure 10 As shown, the device includes: The determination module 1002 is configured to determine the visual features and pixel features corresponding to the original sample image, and process the original sample image using the visual encoder to be trained to obtain basic features; The construction module 1004 is configured to construct visual semantic features based on the visual features and the basic features, and to construct pixel semantic features based on the pixel features and the basic features; The decoding module 1006 is configured to fuse the visual semantic features and the pixel semantic features, and use a pixel decoder to decode the fusion result to obtain the reconstructed image corresponding to the original sample image. The training module 1008 is configured to train the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder.

[0188] In an optional embodiment, determining the visual features and pixel features corresponding to the original sample image includes: The original sample image is input to a visual encoder for encoding processing to obtain visual features; the original sample image is input to a pixel encoder for encoding processing to obtain pixel features; wherein, the visual encoder is a pre-trained visual encoder used to extract global semantic features of the original sample image; the pixel encoder is a pixel-level encoder used to extract detailed semantic features of the original sample image.

[0189] In an optional embodiment, constructing visual semantic features based on the visual features and the basic features includes: The basic features are input into the semantic encoding module for semantic extraction to obtain semantic features; visual semantic features are constructed based on the visual features and the semantic features.

[0190] In an optional embodiment, constructing visual semantic features based on the visual features and the semantic features includes: The semantic features are distilled and aligned with the visual features, and the distilled and aligned semantic features are then dimensionality-reduced using a visual dimensionality reduction module to obtain visual semantic features; wherein, the step of constructing pixel semantic features based on the pixel features and the basic features includes: performing dimensionality reduction on the basic features using a pixel dimensionality reduction module to obtain initial pixel semantic features; and distilling and aligning the initial pixel semantic features with the pixel features to obtain pixel semantic features.

[0191] In an optional embodiment, fusing the visual semantic features and the pixel semantic features, and decoding the fusion result using a pixel decoder to obtain the reconstructed image corresponding to the original sample image, includes: The visual semantic features and the pixel semantic features are concatenated, and the concatenated semantic features are mapped using a fusion module to obtain the target semantic features. The target semantic features are then input into a pixel decoder for decoding to obtain the reconstructed image corresponding to the original sample image.

[0192] In an optional embodiment, training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder includes: The pixel difference loss value of the original sample image and the reconstructed image in the pixel difference dimension is determined according to the reconstruction loss function; the feature loss value of the original sample image and the reconstructed image in the feature dimension is determined according to the perceptual loss function; the semantic loss value of the intermediate features in the semantic stream of the original sample image and the visual features in the semantic dimension is determined according to the semantic distillation loss function; the pixel loss value of the intermediate features in the pixel stream of the original sample image and the pixel features in the pixel dimension is determined according to the pixel distillation loss function; the generation loss value of the original sample image and the reconstructed image in the image generation dimension is determined according to the adversarial loss function; the visual encoder to be trained is trained based on the pixel difference loss value, the feature loss value, the semantic loss value, the pixel loss value, and the generation loss value until a target visual encoder that meets the training stopping condition is obtained.

[0193] In an optional embodiment, after the step of training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder is performed, the method further includes: A diffusion generation model is obtained by training a diffusion generation model based on the target visual encoder; wherein, the target diffusion generation model is used to perform local or global editing processing on the input image.

[0194] In an optional embodiment, training the diffusion generation model based on the target visual encoder to obtain the target diffusion generation model includes: The process involves: acquiring an edit sample image and image editing information; encoding the image editing information using a text segmenter to obtain text features; encoding the edit sample image using the target visual encoder to obtain image semantic features; constructing a context sequence based on the text features and image semantic features; inputting the context sequence and preset noise features into a diffusion generation model for processing to obtain an edit prediction image; and optimizing the diffusion generation model based on the edit prediction image and the edit label image corresponding to the edit sample image to obtain a target diffusion generation model.

[0195] In an optional embodiment, constructing the context sequence based on the text features and the image semantic features includes: Determine the query embedding features; input the query embedding features, the text features, and the image semantic features into the target language model for processing to obtain multimodal hidden state features and target query embedding features; use a connector to process the multimodal hidden state features and the target query embedding features to obtain a context sequence.

[0196] The above is a schematic representation of the image processing apparatus of this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0197] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 11 A schematic diagram of the structure of a second image processing apparatus provided in one embodiment of this specification is shown. Figure 11 As shown, the device includes: The feature determination module 1102 is configured to determine the visual features and pixel features corresponding to the original sample image, and process the original sample image using the visual encoder to be trained to obtain basic features; The feature construction module 1104 is configured to construct visual semantic features based on the visual features and the basic features, and to construct pixel semantic features based on the pixel features and the basic features; The image decoding module 1106 is configured to fuse the visual semantic features and the pixel semantic features, and use a pixel decoder to decode the fusion result to obtain the reconstructed image corresponding to the original sample image; The optimization model module 1108 is configured to train the visual encoder to be trained based on the original sample image and the reconstructed image, and optimize the diffusion generation model based on the trained visual encoder to obtain the target diffusion generation model.

[0198] The above is a schematic representation of the image processing apparatus of this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0199] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 12 A schematic diagram of the structure of a third image processing apparatus provided in one embodiment of this specification is shown. For example... Figure 12 As shown, the device includes: The image acquisition module 1202 is configured to acquire an image to be processed and identify the image region to be edited in the image to be processed; The editing module 1204 is configured to use a target diffusion generation model to edit the image region to be edited in the image to be processed, thereby obtaining a target image corresponding to the image to be processed, wherein the target diffusion generation model is determined by the above method.

[0200] The above is a schematic representation of the image processing apparatus of this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0201] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 13 A schematic diagram of the structure of a fourth image processing apparatus provided in one embodiment of this specification is shown. For example... Figure 13 As shown, the device includes: The image receiving module 1302 is configured to receive an image to be edited and determine the editing information corresponding to the image to be edited; The editing processing module 1304 is configured to use a target diffusion generation model to edit the image to be edited according to the editing information to obtain a target image corresponding to the image to be edited, wherein the target diffusion generation model is determined by the above method.

[0202] The above is a schematic representation of the image processing apparatus of this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0203] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 14 A schematic diagram of the structure of a fifth image processing apparatus provided in one embodiment of this specification is shown. Figure 14 As shown, the device includes: The image receiving module 1402 is configured to receive an image to be enhanced and determine sample generation information corresponding to the image to be enhanced; The editing module 1404 is configured to use a target diffusion generation model to edit the image to be enhanced according to the sample generation information to obtain a sample image corresponding to the image to be enhanced, wherein the target diffusion generation model is determined by the above method.

[0204] The above is a schematic representation of the image processing apparatus of this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0205] Figure 15 A structural block diagram of a computing device 1500 according to one embodiment of this specification is shown. The components of the computing device 1500 include, but are not limited to, a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 via a bus 1530, and a database 1550 is used to store data.

[0206] The computing device 1500 also includes an access device 1540, which enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0207] In one embodiment of this specification, the above-described components of the computing device 1500 and Figure 15 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 15 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0208] The computing device 1500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1500 can also be a mobile or stationary server.

[0209] The processor 1520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0210] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image processing method described above.

[0211] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0212] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image processing method described above.

[0213] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0214] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image processing method described above.

[0215] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0216] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0217] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0218] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0219] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. An image processing method, characterized in that, include: The visual features and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain basic features; Visual semantic features are constructed based on the visual features and the basic features, and pixel semantic features are constructed based on the pixel features and the basic features; The visual semantic features and the pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image; The target visual encoder is obtained by training the visual encoder to be trained based on the original sample image and the reconstructed image.

2. The image processing method according to claim 1, characterized in that, The determination of the visual features and pixel features corresponding to the original sample image includes: The original sample image is input into a visual encoder for encoding processing to obtain visual features; The original sample image is input into a pixel encoder for encoding processing to obtain pixel features; The visual encoder is a pre-trained visual encoder used to extract global semantic features from the original sample image; the pixel encoder is a pixel-level encoder used to extract detailed semantic features from the original sample image.

3. The image processing method according to claim 1, characterized in that, The step of constructing visual semantic features based on the visual features and the basic features includes: The basic features are input into the semantic encoding module for semantic extraction to obtain semantic features; Visual semantic features are constructed based on the visual features and the semantic features.

4. The image processing method according to claim 3, characterized in that, The step of constructing visual semantic features based on the visual features and the semantic features includes: The semantic features and the visual features are distilled and aligned, and the dimensionality reduction of the distilled and aligned semantic features is performed by a visual dimensionality reduction module to obtain visual semantic features. The step of constructing pixel semantic features based on the pixel features and the basic features includes: The basic features are reduced in dimensionality using a pixel dimensionality reduction module to obtain initial pixel semantic features. The initial pixel semantic features are distilled and aligned with the pixel features to obtain the pixel semantic features.

5. The image processing method according to claim 1, characterized in that, The step of fusing the visual semantic features and the pixel semantic features, and then decoding the fusion result using a pixel decoder to obtain the reconstructed image corresponding to the original sample image, includes: The visual semantic features and the pixel semantic features are concatenated, and the concatenated semantic features are mapped using a fusion module to obtain the target semantic features. The target semantic features are input into a pixel decoder for decoding to obtain the reconstructed image corresponding to the original sample image.

6. The image processing method according to claim 1, characterized in that, The step of training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder includes: The pixel difference loss values ​​corresponding to the pixel difference dimension of the original sample image and the reconstructed image are determined according to the reconstruction loss function; The feature loss values ​​of the original sample image and the reconstructed image in the feature dimension are determined according to the perceptual loss function; The semantic loss values ​​of the intermediate features of the original sample image in the semantic stream and the corresponding semantic loss values ​​of the visual features in the semantic dimension are determined based on the semantic distillation loss function. The intermediate features of the original sample image in the pixel stream and the pixel loss value corresponding to the pixel features in the pixel dimension are determined based on the pixel distillation loss function. The generation loss values ​​for the original sample image and the reconstructed image in the image generation dimension are determined based on the adversarial loss function; The visual encoder to be trained is trained based on the pixel difference loss value, the feature loss value, the semantic loss value, the pixel loss value, and the generation loss value until a target visual encoder that meets the training stopping condition is obtained.

7. The image processing method according to any one of claims 1 to 6, characterized in that, After the step of training the visual encoder to be trained based on the original sample image and the reconstructed image to obtain the target visual encoder is executed, the method further includes: A diffusion generation model is obtained by training a diffusion generation model based on the target visual encoder. The target diffusion generation model is used to perform local or global editing on the input image.

8. The image processing method according to claim 7, characterized in that, The step of training a diffusion generation model based on the target visual encoder to obtain a target diffusion generation model includes: Obtain the sample image for editing and image editing information; The image editing information is encoded using a text segmenter to obtain text features, and the edited sample image is encoded using the target visual encoder to obtain image semantic features. A context sequence is constructed based on the text features and the image semantic features, and the context sequence and preset noise features are input into the diffusion generation model for processing to obtain the edit prediction image; The diffusion generation model is optimized based on the edit prediction image and the edit label image corresponding to the edit sample image to obtain the target diffusion generation model.

9. The image processing method according to claim 8, characterized in that, The step of constructing a context sequence based on the text features and the image semantic features includes: Determine the query embedding features; The query embedding features, the text features, and the image semantic features are input into the target language model for processing to obtain multimodal hidden state features and target query embedding features; The context sequence is obtained by processing the multimodal hidden state features and the target query embedding features using a connector.

10. An image processing method, characterized in that, include: The visual features and pixel features corresponding to the original sample image are determined, and the original sample image is processed using the visual encoder to be trained to obtain basic features; Visual semantic features are constructed based on the visual features and the basic features, and pixel semantic features are constructed based on the pixel features and the basic features; The visual semantic features and the pixel semantic features are fused together, and the fusion result is decoded using a pixel decoder to obtain the reconstructed image corresponding to the original sample image; The visual encoder to be trained is trained based on the original sample image and the reconstructed image, and the diffusion generation model is optimized based on the trained visual encoder to obtain the target diffusion generation model.

11. An image processing method, characterized in that, include: Acquire the image to be processed and identify the image region to be edited in the image to be processed; The target image is obtained by editing the image region to be edited in the image to be processed using a target diffusion generation model, wherein the target diffusion generation model is determined by the method described in claim 10.

12. An image processing method, characterized in that, include: Receive the image to be edited and determine the editing information corresponding to the image to be edited; The target diffusion generation model is used to edit the image to be edited according to the editing information to obtain the target image corresponding to the image to be edited, wherein the target diffusion generation model is determined by the method of claim 10.

13. An image processing method, characterized in that, include: Receive the image to be enhanced and determine the sample generation information corresponding to the image to be enhanced; The image to be enhanced is edited using a target diffusion generation model according to the sample generation information to obtain a sample image corresponding to the image to be enhanced, wherein the target diffusion generation model is determined by the method described in claim 10.

14. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images