Image processing method, apparatus, device, medium, and program product

Abstract

The present disclosure relates to an image processing method, device, equipment, medium and program product. The image processing method comprises: generating initial hidden vectors corresponding to multiple frames of original format images respectively, the initial hidden vectors being used to represent image features of the original format images; and generating target images corresponding to the multiple frames of original format images based on the initial hidden vectors and a preset diffusion model, so that the imaging quality of the target images is higher than that of the original format images. The initial hidden vectors and the preset diffusion model are used as the basis for generating the target images, the compressed information of the hidden vectors and the characteristics of the diffusion model for accurately eliminating picture quality rollback are fully utilized, the requirement for computing power is reduced, the imaging quality improvement effect and the adaptability of different products and scenes are ensured, and the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, specifically to an image processing method, apparatus, device, medium, and program product. Background Technology

[0002] In recent years, with the rapid development of image processing technology and the continuous iteration of hardware, electronic devices with image processing and generation functions have been widely used in people's daily lives and work. In specific scenarios, such as at night, it is necessary to perform image processing on the initial image to improve the final image quality.

[0003] However, the image processing technology used is difficult to effectively compensate for the loss of image quality caused by image degradation. Moreover, the processing process requires high computing power, and there are problems such as limited image quality improvement and insufficient applicability, resulting in a poor user experience. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, device, medium, and program product.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, the image processing method comprising:

[0006] Generate initial latent vectors corresponding to multiple frames of original format images, wherein the initial latent vectors are used to characterize the image features of the original format images;

[0007] Based on the initial latent vectors and the preset diffusion model, a target image corresponding to the multiple frames of original format images is generated, so that the imaging quality of the target image is higher than that of the original format image.

[0008] In some embodiments of this disclosure, generating the initial latent vectors corresponding to the multiple frames of original format images includes:

[0009] For each of the original format images, the original format image is input into a preset encoder to convert the original format image to the latent space, thereby obtaining the initial latent vector corresponding to the original format image;

[0010] The step of generating the target image corresponding to the multi-frame original format image based on each of the initial latent vectors and the preset diffusion model includes:

[0011] Based on the initial latent vectors and the preset diffusion model, a target latent vector is generated;

[0012] The target latent vector is input into a preset decoder to obtain the target image corresponding to the target latent vector.

[0013] In some embodiments of this disclosure, generating the target latent vector based on each of the initial latent vectors and the preset diffusion model includes:

[0014] Based on the initial latent vectors and the preset diffusion model, a fused latent vector is generated;

[0015] The target latent vector is generated based on the fused latent vector and the preset diffusion model.

[0016] In some embodiments of this disclosure, generating a fused latent vector based on each of the initial latent vectors and the preset diffusion model includes:

[0017] Degradation detection is performed on each of the original format images to obtain the degradation type corresponding to each of the original format images;

[0018] Based on the degradation type, a first diffusion process is performed on each of the initial latent vectors through the preset diffusion model to obtain the degradation elimination latent vectors corresponding to each of the initial latent vectors.

[0019] Correlation detection is performed on each of the degradation elimination latent vectors to determine the abnormal latent vectors among them.

[0020] The abnormal latent vectors in each of the aforementioned degradation elimination latent vectors are excluded, and the retained degradation elimination latent vectors are fused to obtain the fused latent vector.

[0021] In some embodiments of this disclosure, the step of performing a first diffusion process on each of the initial latent vectors based on the degradation type and the preset diffusion model includes:

[0022] When the original format image corresponds to a degradation type, the initial latent vector corresponding to the original format image, the first random noise, and the degradation type are input into the preset diffusion model to obtain a degradation elimination latent vector corresponding to the initial latent vector;

[0023] When the original format image corresponds to multiple degradation types, each degradation type, the initial latent vector, and the first random noise corresponding to the original format image are respectively input into the preset diffusion model to obtain multiple degradation elimination latent vectors corresponding to the initial latent vector, and each degradation elimination latent vector corresponds to one degradation type.

[0024] In some embodiments of this disclosure, the degradation type includes at least one of noise, blur, low contrast, color distortion, halo, glare, high dynamic range problems, fogging effect, and motion blur.

[0025] In some embodiments of this disclosure, generating the target latent vector based on the fused latent vector and the preset diffusion model includes:

[0026] The preset diffusion model is used to perform multiple diffusion processes. In the first diffusion process, the input of the preset diffusion model includes the fusion latent vector, and in each of the subsequent diffusion processes, the input of the preset diffusion model includes the output result of the preset diffusion model in the previous diffusion process.

[0027] In some embodiments of this disclosure, the step of performing multiple diffusion processes through the preset diffusion model includes:

[0028] By applying the preset diffusion model to the fused latent vector, a second diffusion process is performed to obtain the intermediate latent vector;

[0029] The target latent vector is obtained by performing a third diffusion process on the intermediate latent vector using the preset diffusion model.

[0030] In some embodiments of this disclosure, the image processing method further includes:

[0031] The target category information, target pose information, and target color information of the original format image are determined. The target category information is used to characterize the type of the target object and the image region in which it is located. The target pose information is used to characterize the pose of the target object. The target color information is used to characterize the color of the target object.

[0032] The step of performing a second diffusion process on the fused latent vector through the preset diffusion model to obtain an intermediate latent vector includes:

[0033] The fused latent vector, the target category information, and the second random noise are input into the preset diffusion model to obtain the intermediate latent vector;

[0034] The step of performing a third diffusion process on the intermediate latent vector through the preset diffusion model to obtain the target latent vector includes:

[0035] The intermediate latent vector, the target pose information, the target color information, and the third random noise are input into the preset diffusion model to obtain the target latent vector.

[0036] In some embodiments of this disclosure, determining the target category information, target pose information, and target color information of the original format image includes:

[0037] The original format image is input into a preset target detection model to obtain the target category information;

[0038] Based on the target category information, an image of the area where the target object is located is obtained;

[0039] The region image is input into a preset image-to-text model to obtain the target pose information and the target color information.

[0040] In some embodiments of this disclosure, the training process of the preset diffusion model, preset encoder, and preset decoder includes a first training process and a second training process.

[0041] The first training process includes:

[0042] Based on the original format sample images and the corresponding label images, the initial diffusion model is trained to obtain the intermediate diffusion model;

[0043] The second training process includes:

[0044] The intermediate diffusion model is trained based on the original format sample image and the corresponding label image to obtain the preset diffusion model.

[0045] During the training of the intermediate diffusion model, the weight parameters of the preset encoder and the preset decoder are adjusted.

[0046] In some embodiments of this disclosure, the training process of the preset diffusion model, preset encoder, and preset decoder further includes:

[0047] The original format sample image is subjected to neural network super-resolution processing to obtain the label image.

[0048] In some embodiments of this disclosure, adjusting the weight parameters of the preset encoder and the preset decoder includes:

[0049] During the training of the intermediate diffusion model, the intermediate sample image and / or output sample image are compared with the label image to obtain difference information. The intermediate sample image is obtained by decoding the intermediate latent vector generated by the intermediate diffusion model through the preset decoder, and the output sample image is obtained by decoding the target latent vector generated by the intermediate diffusion model through the preset decoder. The difference information is used to characterize the degree of difference between the intermediate sample image and / or output sample image and the label image.

[0050] Based on the difference information, the weight parameters of the preset encoder and the preset decoder are adjusted to reduce the degree of difference.

[0051] In some embodiments of this disclosure, the consistency comparison between the intermediate sample image and / or the output sample image and the label image includes:

[0052] Perform at least one preset detection on the label image, the intermediate sample image, and / or the output sample image to obtain corresponding detection information;

[0053] The difference information is determined based on the detection information corresponding to the label image, the intermediate sample image, and / or the output sample image.

[0054] In some embodiments of this disclosure, the preset detection includes at least one of target category detection, text detection, face detection, and texture detection, and the detection information corresponding to the target category detection, text detection, face detection, and texture detection is target category information, text information, face information, and texture information, respectively.

[0055] In some embodiments of this disclosure, adjusting the weight parameters of the preset encoder and the preset decoder based on the difference information includes:

[0056] In the first training phase, the weight parameters of the preset encoder and the preset decoder are adjusted based on the difference information determined by the target category information.

[0057] In the second training phase, the weight parameters of the preset encoder and the preset decoder are adjusted based on the difference information determined by the target category information, text information and face information.

[0058] In the third training phase, the weight parameters of the preset encoder and the preset decoder are adjusted based on the difference information determined by the target category information, text information, face information and texture information.

[0059] According to a second aspect of the present disclosure, an image processing apparatus is provided, the image processing apparatus comprising:

[0060] A first generation module is used to generate initial latent vectors corresponding to multiple frames of original format images, and the initial latent vectors are used to characterize the image features of the original format images.

[0061] The second generation module is used to generate a target image corresponding to the multi-frame original format image based on each of the initial latent vectors and the preset diffusion model, so that the imaging quality of the target image is higher than that of the original format image.

[0062] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0063] processor;

[0064] Memory used to store processor-executable instructions;

[0065] The processor is configured to perform the image processing method as described in the first aspect.

[0066] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in the first aspect.

[0067] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method as described in the first aspect.

[0068] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by generating initial latent vectors corresponding to multiple frames of original format images, and generating target images corresponding to multiple frames of original format images based on each initial latent vector and a preset diffusion model, the imaging quality of the target images can be higher than that of the original format images, thus improving the imaging quality. Using the initial latent vectors and the preset diffusion model as the basis for generating the target images fully utilizes the latent vector compression information and the characteristics of the diffusion model to accurately eliminate image quality regression, reducing computational requirements while ensuring the improvement in imaging quality and adaptability to different products and scenarios, thereby enhancing the user experience.

[0069] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0072] Figure 2 This is a schematic diagram illustrating an image processing method according to an exemplary embodiment.

[0073] Figure 3 This is a flowchart illustrating, according to an exemplary embodiment, the generation of target images corresponding to multiple frames of original format images based on each initial latent vector and a preset diffusion model.

[0074] Figure 4 This is a flowchart illustrating, according to an exemplary embodiment, the generation of a target latent vector based on each initial latent vector and a preset diffusion model.

[0075] Figure 5This is a flowchart illustrating, according to an exemplary embodiment, the generation of fused latent vectors based on each initial latent vector and a preset diffusion model.

[0076] Figure 6 This is a flowchart illustrating, according to an exemplary embodiment, the execution of multiple diffusion processes using a preset diffusion model.

[0077] Figure 7 This is a flowchart illustrating, according to an exemplary embodiment, the determination of target category information, target pose information, and target color information of an original format image.

[0078] Figure 8 This is a flowchart illustrating the adjustment of weight parameters of a preset encoder and a preset decoder according to an exemplary embodiment.

[0079] Figure 9 This is a flowchart illustrating a consistency comparison between intermediate sample images and / or output sample images and label images, according to an exemplary embodiment.

[0080] Figure 10 This is a flowchart illustrating, according to an exemplary embodiment, the adjustment of weight parameters of a preset encoder and a preset decoder based on difference information.

[0081] Figure 11 This is a flowchart illustrating an image processing method according to another exemplary embodiment.

[0082] Figure 12 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0083] Figure 13 This is a block diagram of an electronic device according to an exemplary embodiment.

[0084] In the picture:

[0085] 10-First generation module; 20-Second generation module; 101-Processing component; 102-Memory; 103-Power component; 104-Multimedia component; 105-Audio component; 106-Input / output interface; 107-Sensor component; 108-Communication component; 109-Processor. Detailed Implementation

[0086] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0087] In recent years, with the rapid development of image processing technology and the continuous iteration of hardware, electronic devices with image processing and image generation functions, such as mobile phones and cameras, have been widely used in people's daily lives. In specific scenarios, such as at night, due to complex lighting conditions, it is necessary to process the initial image using special algorithms to improve the final image quality.

[0088] In related technologies, multiple RGB images with the same or different exposure levels are acquired simultaneously, and noise reduction and information fusion are performed on the multiple images to improve the image's detail retention or dynamic range.

[0089] However, when using related technologies for image processing, various degradation methods of images will produce different image quality regressions, making it difficult to effectively and comprehensively compensate for the image quality loss caused by image degradation. Furthermore, the processing of RGB images requires high computing power, resulting in problems such as limited image quality improvement effects and limitations on applicable products and scenarios. It is impossible to obtain high-quality images, leading to a poor user experience.

[0090] Based on this, an exemplary embodiment of this disclosure provides an image processing method that generates initial latent vectors corresponding to multiple frames of original format images, and generates target images corresponding to the multiple frames of original format images based on each initial latent vector and a preset diffusion model. This enables the target image to have higher imaging quality than the original format image, thus improving image quality. By using the initial latent vectors and the preset diffusion model as the basis for generating the target image, the method fully utilizes the latent vector compression information and the diffusion model's ability to accurately eliminate image quality regression, reducing computational requirements while ensuring the improvement in imaging quality and adaptability to different products and scenarios, thereby enhancing the user experience.

[0091] In one exemplary embodiment, an image processing method is provided, applied to an electronic device, which may include, for example, a mobile phone, a camera, or other device capable of building a built-in algorithm model for image processing. (Reference) Figure 1 As shown, the image processing methods include:

[0092] S100: Generate initial latent vectors corresponding to multiple frames of original format images. The initial latent vectors are used to characterize the image features of the original format images.

[0093] In step S100, the raw format image is an unprocessed image directly generated by the image sensor, i.e., a RAW image, which has high fidelity and flexible post-processing capabilities. When the electronic device is a camera with shooting capabilities, the raw format image can be an image captured by the camera. Latent vectors are vectors in the field of machine learning that represent data in a low-dimensional latent space, i.e., the latent space. They are used to capture key features of the data, compress the data, and remove redundant information to reduce storage and computational overhead.

[0094] In scenarios such as nighttime photography, image processing requires multi-frame algorithms. Electronic devices can generate corresponding initial latent vectors based on multiple original format images to compress information from each frame. These initial latent vectors characterize the image features of each frame, reflecting the image details of the original format images. For example, the generation of initial latent vectors can be achieved through a preset encoder.

[0095] S200. Based on each initial latent vector and a preset diffusion model, generate target images corresponding to multiple frames of original format images, so that the imaging quality of the target images is higher than that of the original format images.

[0096] In step S200, the diffusion model is a generative model that generates data by simulating the diffusion process of data distribution. It can transform the original data into noisy data by gradually adding noise through forward diffusion, and then recover the data from the noise through reverse denoising. It can be used for image generation and restoration, and has the characteristics of high-quality generation, high flexibility, and high diversity.

[0097] Based on the generated initial latent vectors and a pre-trained diffusion model with a preset structure, a target image corresponding to multiple frames of original format images can be generated, resulting in a target image with higher imaging quality than the original format images. For example, the initial latent vectors can be diffused using a preset diffusion model, and the diffusion result can be decoded using a preset decoder to obtain the target image. It should be noted that image quality is affected by various factors such as sharpness, color reproduction, contrast, dynamic range, saturation, and uniformity. The target image generated based on each initial latent vector and the preset diffusion model can have a significant advantage over the original format image in at least one metric.

[0098] It is understandable that converting the original image format into latent vectors can compress the image information while preserving the original information. This facilitates accurate extraction of image features and reduces the computational power required for subsequent processing. Even devices lacking high computing power can perform image processing, making the above image processing method adaptable to different products. Furthermore, leveraging the high generation quality of the preset diffusion model in conjunction with the initial latent vectors to generate the target image allows for improved removal of degradation effects from different images through structural design and training of the preset diffusion model. This ensures the imaging quality of the target image and makes the above image processing method adaptable to different scenarios.

[0099] In this embodiment, by generating initial latent vectors corresponding to multiple frames of original format images, and generating target images corresponding to these original format images based on each initial latent vector and a preset diffusion model, the imaging quality of the target images can be higher than that of the original format images, thus improving the overall imaging quality. Using the initial latent vectors and the preset diffusion model as the basis for generating the target images fully utilizes the latent vector compression information and the diffusion model's ability to accurately eliminate image quality regression, reducing computational requirements while ensuring improved imaging quality and adaptability to different products and scenarios, thereby enhancing the user experience.

[0100] In some embodiments, generating initial latent vectors corresponding to multiple frames of original format images includes: for each original format image, inputting the original format image into a preset encoder to convert the original format image to the latent space, thereby obtaining the initial latent vector corresponding to the original format image.

[0101] An encoder is a neural network model structure that maps high-dimensional data to a low-dimensional latent representation. It captures key features by learning a compressed representation of the data and possesses characteristics of non-linear mapping, end-to-end processing, and high flexibility. For example... Figure 2 As shown, when generating initial latent vectors corresponding to multiple frames of original format images, for each frame of original format image, the original format image is input into a preset encoder, which is an encoder with a preset network structure and parameters. The preset encoder can then output the initial latent vectors, thus providing a basis for the generation of the target image. It should be noted that... Figure 2 The preset diffusion models shown are all the same preset diffusion model, each representing a different usage process of the preset diffusion model, and can output different results according to different inputs.

[0102] like Figure 3 As shown, based on each initial latent vector and a preset diffusion model, target images corresponding to multiple frames of original format images are generated, including:

[0103] S210. Based on each initial latent vector and the preset diffusion model, generate the target latent vector.

[0104] In step S210, a diffusion process is performed on the initial latent vector using a preset diffusion model. This adds noise and information to the initial latent vector to guide the diffusion process. The diffused latent vector is obtained through the forward diffusion and reverse denoising effects of the preset diffusion model, ultimately generating the target latent vector. This target latent vector represents image features that meet image quality requirements and user needs. For example, each initial latent vector can be diffused separately using the preset diffusion model, and the multiple diffusion results can be fused. Then, the fused result can be diffused multiple times using the preset diffusion model to obtain the target latent vector.

[0105] S220. Input the target latent vector into the preset decoder to obtain the target image corresponding to the target latent vector.

[0106] In step S220, the decoder is a neural network model structure capable of mapping low-dimensional latent vectors back to high-dimensional data spaces such as images. It learns this mapping to generate reconstructed original data or new data samples, featuring non-linear mapping, end-to-end processing, and high-quality generation. Figure 2 As shown, when generating the target image, the target latent vector is input into a preset decoder, which is a decoder with a preset network structure and parameters. The target image can then be generated through the preset decoder. Since the target latent vector can represent image features that meet image quality requirements and user needs, the target image generated by the target latent vector has higher imaging quality than the original format image.

[0107] In this embodiment, by inputting the original format image into a preset encoder, the original format image can be converted into the latent space, realizing the generation of initial latent vectors and providing a basis for the generation of the target image. Based on the initial latent vectors and the preset diffusion model, a target latent vector is generated and input into a preset decoder to obtain the target image corresponding to the target latent vector, thus improving image quality. Through the preset encoder and preset decoder, the conversion between the image space and the low-dimensional latent space can be achieved quickly and accurately. This facilitates the use of the latent vector compression information, providing input to the preset diffusion model and generating the target image based on the output. This further ensures the improved image quality and adaptability to different products and scenarios, enhancing the user experience.

[0108] In some embodiments, reference Figure 4 As shown, based on the initial latent vectors and the preset diffusion model, the target latent vector is generated, including:

[0109] S211. Generate fused latent vectors based on each initial latent vector and the preset diffusion model.

[0110] In step S211, a fused latent vector can be generated based on multiple initial latent vectors and a preset diffusion model. This fused latent vector represents the image features after the fusion of multiple original format images. Since the fused latent vector achieves the fusion of multiple diffusion results, it further compresses the image features and information used subsequently, reducing computational requirements. For example, as shown... Figure 2 As shown, for example, each initial latent vector can be diffused using a preset diffusion model, and the diffusion results can be filtered and fused to obtain a fused latent vector. Thus, the original image features can be accurately restored through the diffusion process, and the restored image features can be compressed through the fusion process.

[0111] S212. Generate the target latent vector based on the fused latent vector and the preset diffusion model.

[0112] In step S212, after obtaining the fused latent vector, it can be diffused using a preset diffusion model. This adds noise and information to the fused latent vector to guide the diffusion process, thereby generating the target latent vector. For example, as shown... Figure 2 As shown, for example, multiple diffusion processes can be performed on the fused latent vector through a preset diffusion model to inject guidance or correction information at different levels through continuous diffusion processes, so as to ensure the accuracy of the target latent vector.

[0113] In this embodiment, a fused latent vector is generated based on each initial latent vector and a preset diffusion model. Then, a target latent vector is generated based on the fused latent vector and the preset diffusion model, thus generating the target latent vector and providing a basis for generating the target image. By generating the fused latent vector, the initial diffusion results are fused. Building upon the information compression characteristics of latent vectors, the fusion of latent vectors further compresses the image features and information used subsequently, reducing computational requirements, ensuring adaptability to different products and scenarios, and improving the user experience.

[0114] As mentioned earlier, due to the influence of the shooting process and conditions, the original format image has corresponding degradation types. Different degradation types can affect the sharpness, color reproduction, contrast, dynamic range, saturation, uniformity, etc. of the original format image, making the imaging quality of the original format image unable to meet the user's needs.

[0115] To address the above problems, in some embodiments, reference is made to... Figure 5 As shown, based on the initial latent vectors and the preset diffusion model, a fused latent vector is generated, including:

[0116] S211-1. Perform degradation detection on each original format image to obtain the corresponding degradation type for each original format image.

[0117] In step S211-1, to compensate for the image quality loss caused by image degradation, it is necessary to understand the degradation type corresponding to each original format image. Therefore, degradation detection is performed on each frame of the original format image to obtain one or more degradation types corresponding to each original format image. For example, each original format image can be input into a pre-trained degradation type detection model, which can then output a corresponding result token based on the original format image, thereby characterizing the degradation type corresponding to the original format image through the result token.

[0118] S211-2. Based on the degradation type, the first diffusion process is performed on each initial latent vector through a preset diffusion model to obtain the degradation elimination latent vector corresponding to each initial latent vector.

[0119] In step S211-2, for each initial latent vector, the degradation type of the original format image corresponding to the initial latent vector is used as a guide. A first diffusion process is performed on the initial latent vector through a preset diffusion model to perform corresponding degradation elimination for the degradation type, resulting in a degradation-eliminated latent vector corresponding to the initial latent vector. This ensures that the degradation-eliminated latent vector is the latent vector of the initial latent vector after eliminating the degradation of that type of image degradation, and can characterize the image features of the original format image after the image degradation is repaired, i.e., the image features corresponding to the real scene. For example, if the degradation type corresponding to a frame of original format image is motion blur, performing the first diffusion process on the initial latent vector corresponding to the original format image through the preset diffusion model can obtain the degradation-eliminated latent vector after eliminating motion blur.

[0120] S211-3. Perform correlation detection on each degradation elimination latent vector to identify the abnormal latent vectors in each degradation elimination latent vector.

[0121] In step S211-3, after obtaining the degenerate elimination latent vectors corresponding to each initial latent vector, if a degenerate elimination latent vector differs significantly from the others, it indicates that the first diffusion process, i.e., the degenerate elimination process, of that degenerate elimination latent vector has deviated, failing to meet the requirements for generating the fused latent vector. Therefore, correlation detection is performed on each degenerate elimination latent vector to identify abnormal latent vectors that differ excessively from others. For example, correlation detection can be performed on each degenerate elimination latent vector using methods such as cosine similarity calculation.

[0122] S211-4. Exclude the abnormal latent vectors in each degradation elimination latent vector, and fuse the retained degradation elimination latent vectors to obtain the fused latent vector.

[0123] In step S211-4, after determining the abnormal latent vectors, the abnormal latent vectors in each degradation elimination latent vector are excluded, and the remaining degradation elimination latent vectors that meet the correlation requirements are fused to realize the generation of fused latent vectors, which further compresses the image features and information used subsequently and reduces the computing power requirements.

[0124] In this embodiment, degradation detection is performed on each original format image to obtain the corresponding degradation type. Based on the degradation type, a first diffusion process is performed on each initial latent vector using a preset diffusion model to obtain the corresponding degradation-removed latent vector. This achieves the removal of image degradation, restores the original image features, and provides a basis for improving image quality. Correlation detection is performed on each degradation-removed latent vector to identify and exclude abnormal latent vectors. The remaining degradation-removed latent vectors are then fused to obtain a fused latent vector. This achieves the fusion of diffusion results (i.e., the degradation-removed latent vectors that meet the requirements), further compressing the image features and information used subsequently, reducing computational requirements, ensuring adaptability to different products and scenarios, and improving user experience.

[0125] In some embodiments, based on the degradation type, a first diffusion process is performed on each initial latent vector using a preset diffusion model, including: when the original format image corresponds to one degradation type, inputting the initial latent vector, first random noise, and degradation type corresponding to the original format image into the preset diffusion model to obtain a degradation-removing latent vector corresponding to the initial latent vector. When the original format image corresponds to multiple degradation types, inputting each degradation type, initial latent vector, and first random noise corresponding to the original format image into the preset diffusion model respectively to obtain multiple degradation-removing latent vectors corresponding to the initial latent vector, each degradation-removing latent vector corresponding to one degradation type.

[0126] When performing the first diffusion process on each initial latent vector according to the degradation type using a preset diffusion model, for each original format image, if the original format image is determined to correspond to a degradation type through degradation detection, then only the image degradation corresponding to one degradation type needs to be eliminated. At this time, the initial latent vector corresponding to the original format image, the first random noise, and this degradation type are used together as input to the preset diffusion model. The preset diffusion model performs one first diffusion process, using the first random noise to provide added noise to the preset diffusion model, and using the degradation type to guide the execution of the first diffusion process, enabling the preset diffusion model to output a degradation elimination latent vector corresponding to the initial latent vector.

[0127] If the original image format is determined to correspond to multiple degradation types through degradation detection, then the image degradation corresponding to each degradation type needs to be eliminated separately. At this point, a preset diffusion model performs multiple first diffusion processes on the initial latent vector. In each first diffusion process, the initial latent vector, first random noise, and one degradation type from the multiple degradation types are used as input to the preset diffusion model. The first random noise provides added noise to the preset diffusion model, and different degradation types provide different insights for each first diffusion process. This allows the preset diffusion model to output multiple degradation elimination latent vectors corresponding to the initial latent vector, with each degradation elimination latent vector corresponding to a degradation type.

[0128] It is understandable that performing the first diffusion process once or multiple times when the original format image corresponds to one or more degradation types, and using a degradation type as a guide in each first diffusion process, can ensure that all degradation types corresponding to the original format image can be accurately eliminated through the first diffusion process, avoiding residual image degradation or erroneous degradation elimination from affecting the final image quality improvement effect.

[0129] In this embodiment, when the original format image corresponds to one degradation type, by inputting the initial latent vector, the first random noise, and the degradation type corresponding to the original format image into a preset diffusion model, a degradation elimination latent vector corresponding to the initial latent vector can be obtained. When the original format image corresponds to multiple degradation types, by inputting each degradation type, the initial latent vector, and the first random noise corresponding to the original format image into the preset diffusion model respectively, multiple degradation elimination latent vectors corresponding to the initial latent vector can be obtained, making each degradation elimination latent vector correspond to one degradation type. This realizes the generation of degradation elimination latent vectors under different conditions, provides a basis for the generation of fused latent vectors, and ensures that all degradation types corresponding to the original format image can be accurately eliminated through the first diffusion process, thereby further ensuring the imaging quality improvement effect.

[0130] In some embodiments, the degradation type includes at least one of noise, blur, low contrast, color distortion, halo, glare, high dynamic range problems, fogging effect, and motion blur.

[0131] Noise, blur, low contrast, color distortion, halo, glare, high dynamic range issues, fogging, and motion blur can be considered as at least one of the following degradation types corresponding to the original format image. Among these, noise is primarily caused by low light intensity in scenes such as nighttime, requiring increased sensitivity to enhance the brightness of the original format image, thus increasing image sensor noise. Noise affects image sharpness and detail, thereby impacting the imaging quality of the original format image and increasing the difficulty of extracting image information.

[0132] Blur includes motion blur, out-of-focus blur, or optical blur. Motion blur is caused by movement of the shooting device or the subject, while out-of-focus blur is due to inaccurate focusing, especially in nighttime scenes where the autofocus system may not function properly. Optical blur is caused by lens quality or improper optical parameter settings. Blur affects image sharpness and detail, thus impacting the image quality of the original format image.

[0133] Low contrast occurs because of uneven light distribution in scenes such as nighttime. Low contrast leads to insufficient contrast between light and dark, resulting in a lack of depth in the image and difficulty in distinguishing between the target and the background, thus affecting the imaging quality of the original format image.

[0134] Color distortion is caused by complex lighting conditions, such as at night. Color distortion can lead to white balance imbalance, resulting in unnatural image colors and affecting the image quality of the original format image.

[0135] Flare and lens flare are caused by the reflection and scattering of strong light sources by the lens, resulting in halos and spots near strong light sources in the image. Flare and lens flare reduce the sharpness and visual effect of the image, thus affecting the imaging quality of the original format image.

[0136] The high dynamic range problem arises because scenes with strong contrast between light and dark areas, such as at night, have a limited dynamic range, which causes the loss of details in both bright and dark areas, thus affecting the image quality of the original format image.

[0137] The fogging effect is caused by the scattering of light by particles in the air, making the image appear blurry and with low contrast, thus affecting the imaging quality of the original format image.

[0138] Motion blur is the trail left by a subject in the frame when the exposure time is long. This can reduce sharpness and affect the image quality of the original format image.

[0139] Understandably, different degradation types will affect the imaging quality of the original format image. By performing degradation detection on the original format image and using the determined degradation type as a guide and basis for performing the first diffusion process, it can be ensured that all degradation types corresponding to the original format image can be accurately eliminated through the first diffusion process. This repairs the image quality loss caused by degradation such as noise, blur, low contrast, color distortion, halo, glare, high dynamic range issues, fogging effect, and motion blur, providing a basis for improving imaging quality.

[0140] In this embodiment, at least one of noise, blur, low contrast, color distortion, halo, glare, high dynamic range problems, fogging effect, and motion blur is taken as the degradation type corresponding to the original format image. Different degradation types can be identified through degradation detection, thereby providing accurate guidance and basis for the first diffusion process. This allows the image quality loss caused by different reasons in the original format image to be repaired, further ensuring the image quality improvement effect and enhancing the user experience.

[0141] In some embodiments, generating a target latent vector based on a fused latent vector and a preset diffusion model includes: performing multiple diffusion processes through the preset diffusion model, wherein the input of the preset diffusion model in the first diffusion process includes the fused latent vector, and the input of the preset diffusion model in each of the remaining diffusion processes includes the output result of the preset diffusion model in the previous diffusion process.

[0142] When generating the target latent vector based on the fused latent vector and the preset diffusion model, multiple diffusion processes are performed through the preset diffusion model. This allows for the injection of guidance or correction information at different levels through multiple diffusion processes. As a result, the target latent vector obtained through multiple diffusion processes can maintain consistency with the original format image as much as possible while eliminating image degradation. This results in a high fidelity target image and further reduces the computational requirements.

[0143] For the first diffusion process in a multi-diffusion process, the fused latent vector is used as the input of the preset diffusion model in this diffusion process. For subsequent diffusion processes, the output of the preset diffusion model in the previous diffusion process is used as the input of the preset diffusion model in this diffusion process. That is, the fused latent vector is first diffused through the preset diffusion model, and then the diffusion result is diffused at least once continuously. The last diffusion result is used as the target latent vector to provide a basis for the generation of the target image.

[0144] In this embodiment, multiple diffusion processes are performed through a preset diffusion model. In the first diffusion process, the input of the preset diffusion model includes the fused latent vector, and in each subsequent diffusion process, the input of the preset diffusion model includes the output result of the preset diffusion model in the previous diffusion process. This achieves multiple and continuous diffusion processes, which facilitates further guidance on the generation of the target latent vector from different levels or factors through multiple diffusion processes. This allows the target latent vector to maintain consistency with the original format image as much as possible while eliminating image degradation, avoiding obvious errors in the image content of the target image generated based on the target latent vector, and further improving the fidelity of the target image.

[0145] In some embodiments, reference Figure 6 As shown, multiple diffusion processes are performed using a preset diffusion model, including:

[0146] S212-1. Using a preset diffusion model, perform a second diffusion process on the fused latent vector to obtain the intermediate latent vector.

[0147] In step S212-1, as follows Figure 2 As shown, by using a preset diffusion model, a second diffusion process is first performed on the fused latent vector to obtain an intermediate latent vector. This second diffusion process can, for example, be used to inject coarser guidance information to avoid significant differences between the target image and the original format image content while achieving degradation reduction and improved imaging quality, thus enhancing the fidelity of the target image.

[0148] S212-2. Using a preset diffusion model, perform a third diffusion process on the intermediate latent vector to obtain the target latent vector.

[0149] In step S212, as Figure 2 As shown, by using a preset diffusion model, a third diffusion process is performed on the intermediate latent vector obtained from the second diffusion process to obtain the target latent vector. The third diffusion process can, for example, be used to inject more refined guidance information to avoid minor differences between the target image and the original format image content while achieving degradation elimination and imaging quality improvement, thereby further improving the fidelity of the target image.

[0150] In this embodiment, a second diffusion process is performed on the fused latent vector using a preset diffusion model to obtain an intermediate latent vector. Then, a third diffusion process is performed on the intermediate latent vector using the same preset diffusion model to obtain the target latent vector. This achieves multiple diffusion processes and the generation of the target latent vector, providing a basis for the generation of the target image. By sequentially executing the second and third diffusion processes, relatively coarse and relatively fine guidance information can be injected in stages, reducing computational requirements. While ensuring that the target image achieves degradation elimination and improved imaging quality, the consistency between the target image and the original format image is maintained, further enhancing the fidelity of the target image.

[0151] In some embodiments, the image processing method further includes: determining target category information, target pose information, and target color information of the original format image, wherein the target category information is used to characterize the type of the target object and the image region in which it is located, the target pose information is used to characterize the pose of the target object, and the target color information is used to characterize the color of the target object.

[0152] As mentioned earlier, when performing the first diffusion process on the initial latent vector, the degradation type and the first random noise are used together with the initial latent vector as input to the preset diffusion model, so as to inject guidance information into the first diffusion process through the degradation type. When performing the second and third diffusion processes through the preset diffusion model, guidance information is also needed for the second and third diffusion processes, so that the second and third diffusion processes can further improve the fidelity of the target image.

[0153] Therefore, before performing the second diffusion process, it is necessary to determine the target category information, target pose information, and target color information of the original format image. The target category information is used as input to the preset diffusion model in the second diffusion process, while the target pose information and target color information are used as input to the preset diffusion model in the third diffusion process. Specifically, the target category information characterizes the type of the target object and its image region in the original format image; that is, the type and location of the main image elements in the original format image. The type of the target object can include, for example, people, animals, buildings, etc. The target pose information characterizes the pose of the target object, such as the relative position and orientation of various body parts of a person. The target color information characterizes the color of the target object.

[0154] By using a preset diffusion model, a second diffusion process is performed on the fused latent vector to obtain an intermediate latent vector. This process includes inputting the fused latent vector, target category information, and second random noise into the preset diffusion model to obtain the intermediate latent vector.

[0155] When performing the second diffusion process on the fused latent vector through the preset diffusion model, the fused latent vector, target category information, and second random noise can be used as inputs to the preset diffusion model. The second random noise provides added noise to the preset diffusion model, and the target category information provides guidance for the preset diffusion model to perform the second diffusion process. This ensures that the image features represented by the intermediate latent vector can match the type of the target object and the image region represented by the target category information, avoiding excessive differences in the type of the target object and the image region corresponding to the original format image, thus improving the fidelity of the target image.

[0156] By using a preset diffusion model, a third diffusion process is performed on the intermediate latent vector to obtain the target latent vector. This process includes inputting the intermediate latent vector, target pose information, target color information, and third random noise into the preset diffusion model to obtain the target latent vector.

[0157] When performing the third diffusion process on the intermediate latent vector through the preset diffusion model, the intermediate latent vector, target pose information, target color, and third random noise can be used as inputs to the preset diffusion model. The third random noise provides added noise to the preset diffusion model, and the target pose information and target color information provide guidance for the preset diffusion model to perform the third diffusion process. This ensures that the image features represented by the target latent vector can match the pose and color of the target object represented by the target pose information and target color information, avoiding excessive differences in pose and color between the subsequently generated target image and the original format image, thus improving the fidelity of the target image.

[0158] In this embodiment, by determining the target category information, target pose information, and target color information of the original format image, input is provided for the preset diffusion model of the second and third diffusion processes. The target category information can guide the second diffusion process, and the target pose information and target color information can guide the third diffusion process. This enables the second and third diffusion processes to perform coarse and fine corrections to the image content, ensuring that the generated target image does not differ too much from the type, position, pose, and color of the object corresponding to the original format image, thereby further improving the fidelity of the target image.

[0159] In some embodiments, reference Figure 7 As shown, the target category information, target pose information, and target color information of the original format image are determined, including:

[0160] S310. Input the original format image into the preset target detection model to obtain target category information.

[0161] In step S310, the pre-trained preset target detection model can be used to detect and extract target category information. By inputting the original format image into the preset target detection model, the target category information of the original format image can be obtained. A lightweight preset target detection model can be used to achieve faster operation and adapt to different products. For example, the preset target detection model may include the YOLO detection model.

[0162] S320: Based on the target category information, obtain an image of the area where the target object is located.

[0163] In step S320, since the target category information can characterize the image region where the target object is located, the original format image can be cropped according to the image region where the target object is located as characterized by the target category information to obtain the image of the region where the target object is located. This facilitates the subsequent extraction of target pose information and target color information within a smaller window, reducing the amount of computation.

[0164] S330. Input the region image into the preset image-to-text model to obtain the target pose information and target color information.

[0165] In step S330, the pre-trained image-to-text model can be used to generate corresponding text information, i.e., semantic extraction, from the image. The text information describes the pose and color of the target object. By inputting a region image containing the target object and smaller than the original format image into the pre-trained image-to-text model, the target pose and color information of the original format image can be obtained. A lightweight image-to-text model can be used to achieve faster semantic extraction.

[0166] In this embodiment, by using a preset target detection model and a preset image-to-text model, target category information, target pose information, and target color information can be output respectively. This enables the determination of target category information, target pose information, and target color information, providing a basis for the second and third diffusion processes. The preset detection model and preset image-to-text model have high stability and running speed, further improving the accuracy and efficiency of generating target images. Using the image of the area where the target object is located as the basis for determining the target pose information and target color information reduces the computational load of this determination process, further reducing the computing power requirements and thus ensuring compatibility with different products.

[0167] In some embodiments, the training process of the preset diffusion model, preset encoder, and preset decoder includes a first training process and a second training process. The first training process includes: training an initial diffusion model based on the original format sample image and the corresponding label image to obtain an intermediate diffusion model.

[0168] As mentioned earlier, the generation of the target image requires the use of a preset diffusion model, a preset encoder, and a preset decoder. Before generating the target image, the preset diffusion model, the preset encoder, and the preset decoder need to be trained to ensure that the diffusion process of the preset diffusion model, the encoding process of the preset encoder, and the decoding process of the preset decoder can meet the requirements for improving image imaging quality.

[0169] The training process for the pre-defined diffusion model, pre-defined encoder, and pre-defined decoder includes a first training process and a second training process. In the first training process, the original format sample images are used as training samples, and the corresponding label images are used as training labels to train the initial diffusion model, resulting in an intermediate diffusion model upon completion of the first training process. To ensure training effectiveness, the label images need to have relatively consistent image content with their corresponding original format sample images, and the image quality of the label images needs to be higher than that of the original format sample images. During the first training process, the weight parameters of the pre-defined encoder and pre-defined decoder need to be kept fixed, and only the pre-defined diffusion model is trained.

[0170] For example, the original format sample image can be input into the initial diffusion model, and the parameters of the initial diffusion model can be adjusted by backpropagating the loss function based on the difference between the output and the labeled image, so as to iteratively train the initial diffusion model. When a certain number of iterations is reached or the difference is less than a certain threshold, the training of the initial diffusion model is stopped, and the first training process is considered to be completed.

[0171] The second training process includes: training the intermediate diffusion model based on the original format sample images and the corresponding label images to obtain the preset diffusion model; and adjusting the weight parameters of the preset encoder and preset decoder during the training process of the intermediate diffusion model.

[0172] The original format sample images can be used as training samples, and the corresponding label images can be used as training labels to train the intermediate diffusion model, so that the preset diffusion model is obtained when the second training process is completed. The training process of the intermediate diffusion model is logically similar to that of the initial diffusion model, and will not be described in detail here.

[0173] During the training of the intermediate diffusion model, the weight parameters of the pre-defined encoder and decoder are adjusted to ensure they have appropriate weights by the end of the second training phase. The weight parameters of the pre-defined encoder may include, for example, convolutional kernel weights that determine how image features are extracted and bias terms used to adjust the activation values ​​of the convolutional output. Adjusting the weight parameters of the pre-defined encoder reflects the adjustment of how it performs feature extraction and dimensionality reduction. Similarly, the weight parameters of the pre-defined decoder may include, for example, deconvolution kernel weights that determine how images are generated from latent vectors and bias terms used to adjust the activation values ​​of the deconvolutional output. Adjusting the weight parameters of the pre-defined decoder reflects the adjustment of how it generates new images.

[0174] In this embodiment, an initial diffusion model is trained based on the original format sample image and the corresponding label image to obtain an intermediate diffusion model, thus realizing the first training process. Then, the intermediate diffusion model is trained based on the original format sample image and the corresponding label image to obtain a preset diffusion model. During the training of the intermediate diffusion model, the weight parameters of the preset encoder and preset decoder are adjusted, thus realizing the second training process. Through the first and second training processes, the diffusion process of the preset diffusion model, the encoding process of the preset encoder, and the decoding process of the preset decoder can meet the requirements for improving image imaging quality, providing a basis for the generation of the target image, ensuring the imaging quality improvement effect and adaptability to different products and scenarios, and enhancing the user experience.

[0175] In some embodiments, the training process of the preset diffusion model, preset encoder, and preset decoder further includes: performing neural network super-resolution processing on the original format sample image to obtain a label image.

[0176] The training process of the preset diffusion model, preset encoder and preset decoder requires the use of original format sample images and their corresponding label images. Original format sample images can be obtained through simulation rendering, real image acquisition or sample set screening.

[0177] As mentioned earlier, to ensure training effectiveness, the label images need to have content that is largely consistent with the corresponding original format sample images, and the image quality of the label images needs to be higher than that of the original format sample images. Therefore, neural network super-resolution processing, namely RAWSR (RAW Image Super-Resolution), can be applied to the original format sample images to convert them from low resolution to high resolution. Conventional processing methods can be used for neural network super-resolution to ensure that the label images guide the training process.

[0178] In this embodiment, label images are obtained by performing neural network super-resolution processing on the original format sample images, thus realizing the generation of label images. This provides a basis for the training of the preset diffusion model, preset encoder, and preset decoder, and ensures that the label images have relatively consistent picture content with the corresponding original format sample images. Furthermore, the imaging quality of the label images needs to be higher than that of the original format sample images, thereby ensuring the guiding significance of the label images for the training process.

[0179] In some embodiments, reference Figure 8 As shown, adjust the weight parameters of the preset encoder and preset decoder, including:

[0180] S410. During the training of the intermediate diffusion model, the intermediate sample image and / or output sample image are compared with the label image to obtain difference information. The intermediate sample image is obtained by decoding the intermediate latent vector generated by the intermediate diffusion model through a preset decoder. The output sample image is obtained by decoding the target latent vector generated by the intermediate diffusion model through a preset decoder. The difference information is used to characterize the degree of difference between the intermediate sample image and / or output sample image and the label image.

[0181] In step S410, during the second training process of the intermediate diffusion model, the intermediate latent vectors generated by the intermediate diffusion model can be decoded by a preset decoder to obtain intermediate sample images in the second training process. The target latent vectors generated by the intermediate diffusion model can also be decoded by a preset decoder to obtain output sample images in the second training process.

[0182] At least one of the intermediate sample image and the output sample image can be compared with the label image to obtain difference information. This difference information can be used to characterize at least one of the differences between the intermediate sample image and the label image and the differences between the output sample image and the label image. This information can represent whether the imaging quality of the intermediate sample image and the output sample image output by the preset diffusion model, preset encoder, and preset decoder at the current training level can reach the level of the label image. This provides a basis for the trend and degree of adjusting the weight parameters of the preset encoder and preset decoder.

[0183] S420. Based on the difference information, adjust the weight parameters of the preset encoder and preset decoder to reduce the degree of difference.

[0184] In step S420, the weight parameters of the preset encoder and preset decoder are adjusted according to the degree of difference represented by the difference information to reduce the degree of difference. This allows the intermediate sample images and output sample images to gradually converge towards the label images, ensuring that the encoding process of the preset encoder and the decoding process of the preset decoder can meet the requirement of improving the imaging quality of the original format sample images to the imaging quality of the label images. Upon completion of the second training process, the weight parameters of the preset encoder and preset decoder can support the generation of target images that meet user requirements based on the actual multiple frames of original format images.

[0185] In this embodiment, during the training of the intermediate diffusion model, at least one of the intermediate sample image and the output sample image is compared with the label image to obtain difference information. Based on the difference information, the weight parameters of the preset encoder and the preset decoder are adjusted to reduce the degree of difference. This realizes the training of the preset encoder and the preset decoder in the second training process, so that the intermediate sample image and the output sample image can gradually approach the label image. This ensures that the trained preset encoder and the preset decoder can generate a high-fidelity target image together with the preset diffusion model, further ensuring the improvement of imaging quality and enhancing the user experience.

[0186] In some embodiments, reference Figure 9 As shown, a consistency comparison is performed between the intermediate sample images and / or the output sample images and the label images, including:

[0187] S411. Perform at least one preset detection on the label image, intermediate sample image and / or output sample image respectively to obtain the corresponding detection information.

[0188] In step S411, one or more preset detections are performed on at least one of the intermediate sample image and the output sample image, as well as the label image, to obtain detection information corresponding to each image and each preset detection. The preset detection is used to detect specific features of the image and characterize the image features through the detection information, providing a basis for subsequently determining the consistency of specific features between images.

[0189] S412. Determine the difference information based on the detection information corresponding to the label image, intermediate sample image, and / or output sample image.

[0190] In step S412, difference information is determined based on the detection information corresponding to the label image and the detection information corresponding to at least one of the intermediate sample image and the output sample image. This difference information is used to determine the degree of difference between the images, thus providing a basis for adjusting the weight parameters of the preset encoder and preset decoder. For example, if the detection information of each image includes the actual value corresponding to a specific image feature, the difference in values ​​can be used as the difference information. If the detection information of an image includes the type corresponding to a specific image feature, the smaller value can be used as the difference information when the types are the same, and the larger value can be used as the difference information when the types are different.

[0191] In this embodiment, at least one preset detection is performed on the label image, intermediate sample image, and / or output sample image to obtain corresponding detection information. Based on the detection information corresponding to the label image, intermediate sample image, and / or output sample image, difference information is determined, thus providing a basis for adjusting the weight parameters of the preset encoder and preset decoder. Using the detection information obtained through preset detection as the basis for determining difference information can provide guidance for the training of the preset encoder and preset decoder in a specific dimension. This helps ensure that the trained preset encoder and preset decoder can support the generation of high-fidelity target images, further guaranteeing the improvement in imaging quality and enhancing the user experience.

[0192] In some embodiments, the preset detection includes at least one of target category detection, text detection, face detection, and texture detection, and the detection information corresponding to target category detection, text detection, face detection, and texture detection is target category information, text information, face information, and texture information, respectively.

[0193] At least one of target category detection, text detection, face detection, and texture detection can be used as a preset detection for intermediate sample images, output sample images, and label images to obtain corresponding detection information.

[0194] Among them, target category detection is used to obtain target category information corresponding to the image. Target category information can characterize the type of the target object in the image and the image region it is located in. Target category detection can be achieved, for example, by inputting the image into a preset target detection model. Using target category information as detection information to determine difference information can restrict image features such as the type of the target object and the image region it is located in. After adjusting the weight parameters of the preset encoder and preset decoder according to the difference information, it can avoid the error that the generated target image has a target object type and image region that are too different from the original format image. The fidelity of the target image can be improved in practical applications during the training process of the preset encoder and preset decoder.

[0195] Text detection and face detection are used to acquire text and face information from images, respectively, representing text content and facial details. Text and face detection can be achieved by inputting the image into a pre-defined text detection model and a pre-defined face detection model. Since both text and face scenes are sensitive, errors are easily detected. Using text or face information as detection data to determine discrepancies can constrain image features such as text content and facial details. By adjusting the weight parameters of the pre-defined encoder and decoder based on these discrepancies, errors can be avoided where the generated target image shows significant differences between the text content or facial details and the original format image. This ensures robustness to both text and face scenes, and improves the fidelity of the target image during the training of the pre-defined encoder and decoder in practical applications.

[0196] Texture detection is used to acquire texture information in an image. Texture information can characterize the texture details in an image. Texture detection can be achieved, for example, by inputting the image into a pre-defined texture detection model. Using texture information as detection information to determine difference information can restrict image features such as texture details. After adjusting the weight parameters of the pre-defined encoder and decoder based on the difference information, dense grid-like textures in the generated target image can be avoided, which would affect the visual effect. The fidelity of the target image can be improved during the training process of the pre-defined encoder and decoder in practical applications.

[0197] In this embodiment, at least one of target category detection, text detection, face detection, and texture detection is used as a preset detection method. This allows for the acquisition of target category information, text information, face information, and texture information, respectively. By using these as detection information to determine the difference information and adjusting the preset encoder and preset decoder based on the difference information, restrictions on image features such as the type of the photographed object and its image region, text content, facial details, and texture details can be added during the training process. This avoids errors in the generation of the target image, thereby improving the fidelity of the subsequent target image during the training of the preset encoder and preset decoder.

[0198] In some embodiments, reference Figure 10 As shown, based on the difference information, the weight parameters of the preset encoder and preset decoder are adjusted, including:

[0199] S421. In the first training phase, the weight parameters of the preset encoder and preset decoder are adjusted based on the difference information determined by the target category information.

[0200] In step S421, the second training process of adjusting the weight parameters of the preset encoder and preset decoder includes a first training phase, a second training phase, and a third training phase. In the first training phase, target category information is used as the basis for determining difference information, and the preset encoder and preset decoder are adjusted according to the difference information. This allows for the inclusion of constraints on the type of the captured object and the image region it occupies in the first training phase, thereby avoiding errors in the target category dimension in the target image generation result.

[0201] S422. In the second training phase, based on the difference information determined by the target category information, text information, and face information, adjust the weight parameters of the preset encoder and preset decoder.

[0202] In step S422, during the second training phase, target category information, text information, and face information are used together as the basis for determining the difference information. The preset encoder and preset decoder are adjusted according to the difference information. This allows for the addition of restrictions on text content and face details in the second training phase, based on the first training phase, to avoid errors in the text and face dimensions of the target image generation result.

[0203] S423. In the third training phase, the weight parameters of the preset encoder and preset decoder are adjusted based on the difference information determined by the target category information, text information, face information and texture information.

[0204] In step S423, during the third training phase, target category information, text information, face information, and texture information are used together as the basis for determining the difference information. The preset encoder and preset decoder are adjusted according to the difference information. This allows for the addition of texture detail constraints in the third training phase, based on the first and second training phases, to avoid errors in texture dimension in the image generation results.

[0205] It should be noted that in the second training phase, when determining the difference information based on target category information, text information, and facial information, the corresponding feature values ​​of target category information, text information, and facial information can be accumulated, and the accumulated result can be used as the difference information for the second training phase. Similarly, in the third training phase, when determining the difference information based on target category information, text information, facial information, and texture information, these three information can be accumulated, and the accumulated result can be used as the difference information for the third training phase. Since people's sensitivity to differences in target category, text, face, and texture gradually decreases, dividing the training into three phases allows for gradual optimization from high sensitivity to low sensitivity and from large scenes to small scenes, ensuring the consistency between intermediate sample images, output sample images, and label images when the second training process is completed.

[0206] In this embodiment, the training process of the preset encoder and preset decoder is divided into three training stages. The weight parameters of the preset encoder and preset decoder are adjusted according to different difference information in the three training stages. This achieves gradual optimization from high sensitivity to low sensitivity and from large scene to small scene, avoiding errors in target category, text, face and texture dimensions in the image generation results. As a result, the fidelity of the subsequent target image is improved during the training process of the preset encoder and preset decoder, further ensuring the imaging quality improvement effect and improving the user experience.

[0207] In one exemplary embodiment, an image processing method is provided, applied to an electronic device, with reference to... Figure 11 As shown, the image processing methods include:

[0208] S1. Based on the original format sample images and the corresponding label images, train the initial diffusion model to obtain the intermediate diffusion model;

[0209] S2. Based on the original format sample images and the corresponding label images, train the intermediate diffusion model to obtain the preset diffusion model, and adjust the weight parameters of the preset encoder and preset decoder.

[0210] S3. For each original format image, input the original format image into the preset encoder to convert the original format image to the latent space and obtain the initial latent vector corresponding to the original format image.

[0211] S4. Perform degradation detection on each original format image to obtain the corresponding degradation type for each original format image;

[0212] S5. When the original format image corresponds to a degradation type, the initial latent vector, the first random noise, and the degradation type corresponding to the original format image are input into the preset diffusion model to obtain a degradation elimination latent vector corresponding to the initial latent vector.

[0213] S6. When the original format image corresponds to multiple degradation types, each degradation type, the initial latent vector, and the first random noise corresponding to the original format image are input into the preset diffusion model to obtain multiple degradation elimination latent vectors corresponding to the initial latent vector, and each degradation elimination latent vector corresponds to a degradation type.

[0214] S7. Perform correlation detection on each degradation elimination latent vector to identify the abnormal latent vectors in each degradation elimination latent vector;

[0215] S8. Exclude the abnormal latent vectors in each degradation elimination latent vector, and fuse the retained degradation elimination latent vectors to obtain the fused latent vector;

[0216] S9. Input the original format image into the preset target detection model to obtain target category information;

[0217] S10. Based on the target category information, obtain an image of the area where the target object is located;

[0218] S11. Input the region image into the preset image-to-text model to obtain the target pose information and target color information;

[0219] S12. Input the fused latent vector, target category information and second random noise into the preset diffusion model to obtain the intermediate latent vector;

[0220] S13. Input the intermediate latent vector, target pose information, target color information and third random noise into the preset diffusion model to obtain the target latent vector;

[0221] S14. Input the target latent vector into the preset decoder to obtain the target image corresponding to the target latent vector.

[0222] In this embodiment, by generating initial latent vectors corresponding to multiple frames of original format images, and generating target images corresponding to these original format images based on each initial latent vector and a preset diffusion model, the imaging quality of the target images can be higher than that of the original format images, thus improving the overall imaging quality. Using the initial latent vectors and the preset diffusion model as the basis for generating the target images fully utilizes the latent vector compression information and the diffusion model's ability to accurately eliminate image quality regression, reducing computational requirements while ensuring improved imaging quality and adaptability to different products and scenarios, thereby enhancing the user experience.

[0223] In one exemplary embodiment, an image processing apparatus is provided, applied to an electronic device, with reference to... Figure 12 As shown, the image processing device includes a first generation module 10 and a second generation module 20. The first generation module 10 is used to generate initial latent vectors corresponding to multiple frames of original format images, and the initial latent vectors are used to characterize the image features of the original format images. The second generation module 20 is used to generate target images corresponding to multiple frames of original format images based on each initial latent vector and a preset diffusion model, so that the imaging quality of the target images is higher than that of the original format images.

[0224] In this embodiment, the first generation module 10 generates initial latent vectors corresponding to multiple frames of original format images, and the second generation module 20 generates target images corresponding to these original format images based on the initial latent vectors and a preset diffusion model. This enables the target images to have higher imaging quality than the original format images, thus improving overall image quality. By using the initial latent vectors and the preset diffusion model as the basis for generating the target images, the system fully utilizes the latent vector compression information and the diffusion model's ability to accurately eliminate image quality regression, reducing computational requirements while ensuring improved imaging quality and adaptability to different products and scenarios, thereby enhancing the user experience.

[0225] In one embodiment, the first generation module 10 is further configured to: input the original format image to a preset encoder for each original format image to convert the original format image to the latent space and obtain the initial latent vector corresponding to the original format image; the second generation module 20 is further configured to: generate a target latent vector based on each initial latent vector and a preset diffusion model; input the target latent vector to a preset decoder to obtain the target image corresponding to the target latent vector.

[0226] In one embodiment, the second generation module 20 is further configured to: generate a fused latent vector based on each initial latent vector and a preset diffusion model; and generate a target latent vector based on the fused latent vector and the preset diffusion model.

[0227] In one embodiment, the second generation module 20 is further configured to: perform degradation detection on each original format image to obtain the degradation type corresponding to each original format image; based on the degradation type, perform a first diffusion process on each initial latent vector through a preset diffusion model to obtain the degradation elimination latent vector corresponding to each initial latent vector; perform correlation detection on each degradation elimination latent vector to determine the abnormal latent vector in each degradation elimination latent vector; exclude the abnormal latent vector in each degradation elimination latent vector, and fuse the retained degradation elimination latent vectors to obtain the fused latent vector.

[0228] In one embodiment, the second generation module 20 is further configured to: when the original format image corresponds to one degradation type, input the initial latent vector, the first random noise, and the degradation type corresponding to the original format image into a preset diffusion model to obtain a degradation elimination latent vector corresponding to the initial latent vector; when the original format image corresponds to multiple degradation types, input each degradation type, the initial latent vector, and the first random noise corresponding to the original format image into the preset diffusion model to obtain multiple degradation elimination latent vectors corresponding to the initial latent vector, each degradation elimination latent vector corresponding to one degradation type.

[0229] In one embodiment, the degradation type includes at least one of noise, blur, low contrast, color distortion, halo, glare, high dynamic range problems, fogging effect, and motion blur.

[0230] In one embodiment, the second generation module 20 is further configured to: perform multiple diffusion processes using a preset diffusion model, wherein the input of the preset diffusion model in the first diffusion process includes the fused latent vector, and the input of the preset diffusion model in each of the remaining diffusion processes includes the output result of the preset diffusion model in the previous diffusion process.

[0231] In one embodiment, the second generation module 20 is further configured to: perform a second diffusion process on the fused latent vector using a preset diffusion model to obtain an intermediate latent vector; and perform a third diffusion process on the intermediate latent vector using the preset diffusion model to obtain a target latent vector.

[0232] In one embodiment, the second generation module 20 is further configured to: determine the target category information, target pose information, and target color information of the original format image, wherein the target category information is used to characterize the type of the target object and the image region in which it is located, the target pose information is used to characterize the pose of the target object, and the target color information is used to characterize the color of the target object; input the fused latent vector, the target category information, and the second random noise into a preset diffusion model to obtain an intermediate latent vector; and input the intermediate latent vector, the target pose information, the target color information, and the third random noise into the preset diffusion model to obtain a target latent vector.

[0233] In one embodiment, the second generation module 20 is further configured to: input the original format image into a preset target detection model to obtain target category information; based on the target category information, obtain an image of the area where the target object is located; input the area image into a preset image-to-text model to obtain target pose information and target color information.

[0234] In one embodiment, the image processing apparatus further includes a training module. The training process of the preset diffusion model, the preset encoder, and the preset decoder includes a first training process and a second training process. The training module is used to: train the initial diffusion model based on the original format sample image and the corresponding label image to obtain an intermediate diffusion model; train the intermediate diffusion model based on the original format sample image and the corresponding label image to obtain a preset diffusion model; and adjust the weight parameters of the preset encoder and the preset decoder during the training process of the intermediate diffusion model.

[0235] In one embodiment, the training module is further configured to: perform neural network super-resolution processing on the original format sample image to obtain a label image.

[0236] In one embodiment, the training module is further configured to: during the training of the intermediate diffusion model, perform a consistency comparison between the intermediate sample image and / or the output sample image and the label image to obtain difference information; the intermediate sample image is obtained by decoding the intermediate latent vector generated by the intermediate diffusion model through a preset decoder; the output sample image is obtained by decoding the target latent vector generated by the intermediate diffusion model through a preset decoder; the difference information is used to characterize the degree of difference between the intermediate sample image and / or the output sample image and the label image; and based on the difference information, adjust the weight parameters of the preset encoder and the preset decoder to reduce the degree of difference.

[0237] In one embodiment, the training module is further configured to: perform at least one preset detection on the label image, intermediate sample image, and / or output sample image respectively to obtain corresponding detection information; and determine difference information based on the detection information corresponding to the label image, intermediate sample image, and / or output sample image.

[0238] In one embodiment, the preset detection includes at least one of target category detection, text detection, face detection, and texture detection, and the detection information corresponding to target category detection, text detection, face detection, and texture detection is target category information, text information, face information, and texture information, respectively.

[0239] In one embodiment, the training module is further configured to: in a first training phase, adjust the weight parameters of the preset encoder and preset decoder based on the difference information determined by the target category information; in a second training phase, adjust the weight parameters of the preset encoder and preset decoder based on the difference information determined by the target category information, text information, and face information; and in a third training phase, adjust the weight parameters of the preset encoder and preset decoder based on the difference information determined by the target category information, text information, face information, and texture information.

[0240] In one exemplary embodiment, an electronic device is provided, which may include, for example, a mobile phone, a camera, or other device capable of having a built-in algorithm model for image processing.

[0241] refer to Figure 13 As shown, the electronic device may include one or more of the following components: processing component 101, memory 102, power component 103, multimedia component 104, audio component 105, input / output (I / O) interface 106, sensor component 107, and communication component 108.

[0242] Processing component 101 typically controls the overall operation of an electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 101 may include one or more processors 109 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 101 may include one or more modules to facilitate interaction between processing component 101 and other components. For example, processing component 101 may include a multimedia module to facilitate interaction between multimedia component 104 and processing component 101.

[0243] Memory 102 is configured to store various types of data to support the operation of the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc. Memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0244] Power component 103 provides power to various components of the electronic device. Power component 103 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0245] Multimedia component 104 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 104 includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0246] Audio component 105 is configured to output and / or input audio signals. For example, audio component 105 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 102 or transmitted via communication component 108. In some embodiments, audio component 105 also includes a speaker for outputting audio signals.

[0247] I / O interface 106 provides an interface between processing component 101 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0248] Sensor assembly 107 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 107 can detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. Sensor assembly 107 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 107 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 107 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0249] Communication component 108 is configured to facilitate wired or wireless communication between electronic devices and other devices. Devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 108 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 108 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0250] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method applied to the electronic device described above.

[0251] In one exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 102 including instructions, which can be executed by a processor 109 of an electronic device to perform the image processing method applied to the electronic device described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the instructions in the storage medium are executed by the processor 109 of the electronic device, the electronic device is able to perform the image processing method shown in the above embodiments.

[0252] In one exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by processor 109, implements the image processing method shown in the above embodiments.

[0253] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0254] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The image processing method includes: Generate initial latent vectors corresponding to multiple frames of original format images, wherein the initial latent vectors are used to characterize the image features of the original format images; Based on the initial latent vectors and the preset diffusion model, a target image corresponding to the multiple frames of original format images is generated, so that the imaging quality of the target image is higher than that of the original format image.

2. The image processing method according to claim 1, characterized in that, The generation of initial latent vectors corresponding to the multiple frames of original format images includes: For each of the original format images, the original format image is input into a preset encoder to convert the original format image to the latent space, thereby obtaining the initial latent vector corresponding to the original format image; The step of generating the target image corresponding to the multi-frame original format image based on each of the initial latent vectors and the preset diffusion model includes: Based on the initial latent vectors and the preset diffusion model, a target latent vector is generated; The target latent vector is input into a preset decoder to obtain the target image corresponding to the target latent vector.

3. The image processing method according to claim 2, characterized in that, The step of generating a target latent vector based on the initial latent vectors and the preset diffusion model includes: Based on the initial latent vectors and the preset diffusion model, a fused latent vector is generated; The target latent vector is generated based on the fused latent vector and the preset diffusion model.

4. The image processing method according to claim 3, characterized in that, The step of generating a fused latent vector based on the initial latent vectors and the preset diffusion model includes: Degradation detection is performed on each of the original format images to obtain the degradation type corresponding to each of the original format images; Based on the degradation type, a first diffusion process is performed on each of the initial latent vectors through the preset diffusion model to obtain the degradation elimination latent vectors corresponding to each of the initial latent vectors. Correlation detection is performed on each of the degradation elimination latent vectors to determine the abnormal latent vectors among them. The abnormal latent vectors in each of the aforementioned degradation elimination latent vectors are excluded, and the retained degradation elimination latent vectors are fused to obtain the fused latent vector.

5. The image processing method according to claim 4, characterized in that, The first diffusion process, based on the degradation type and using the preset diffusion model, is performed on each of the initial latent vectors, including: When the original format image corresponds to a degradation type, the initial latent vector corresponding to the original format image, the first random noise, and the degradation type are input into the preset diffusion model to obtain a degradation elimination latent vector corresponding to the initial latent vector; When the original format image corresponds to multiple degradation types, each degradation type, the initial latent vector, and the first random noise corresponding to the original format image are respectively input into the preset diffusion model to obtain multiple degradation elimination latent vectors corresponding to the initial latent vector, and each degradation elimination latent vector corresponds to one degradation type.

6. The image processing method according to claim 4, characterized in that, The degradation types include at least one of noise, blur, low contrast, color distortion, halo, glare, high dynamic range problems, fogging effect, and motion blur.

7. The image processing method according to claim 3, characterized in that, The step of generating the target latent vector based on the fused latent vector and the preset diffusion model includes: The preset diffusion model is used to perform multiple diffusion processes. In the first diffusion process, the input of the preset diffusion model includes the fusion latent vector, and in each of the subsequent diffusion processes, the input of the preset diffusion model includes the output result of the preset diffusion model in the previous diffusion process.

8. The image processing method according to claim 7, characterized in that, The process of performing multiple diffusion processes through the preset diffusion model includes: By applying the preset diffusion model to the fused latent vector, a second diffusion process is performed to obtain the intermediate latent vector; The target latent vector is obtained by performing a third diffusion process on the intermediate latent vector using the preset diffusion model.

9. The image processing method according to claim 8, characterized in that, The image processing method further includes: The target category information, target pose information, and target color information of the original format image are determined. The target category information is used to characterize the type of the target object and the image region in which it is located. The target pose information is used to characterize the pose of the target object. The target color information is used to characterize the color of the target object. The step of performing a second diffusion process on the fused latent vector through the preset diffusion model to obtain an intermediate latent vector includes: The fused latent vector, the target category information, and the second random noise are input into the preset diffusion model to obtain the intermediate latent vector; The step of performing a third diffusion process on the intermediate latent vector through the preset diffusion model to obtain the target latent vector includes: The intermediate latent vector, the target pose information, the target color information, and the third random noise are input into the preset diffusion model to obtain the target latent vector.

10. The image processing method according to claim 9, characterized in that, Determining the target category information, target pose information, and target color information of the original format image includes: The original format image is input into a preset target detection model to obtain the target category information; Based on the target category information, an image of the area where the target object is located is obtained; The region image is input into a preset image-to-text model to obtain the target pose information and the target color information.

11. The image processing method according to any one of claims 2 to 10, characterized in that, The training process of the preset diffusion model, preset encoder, and preset decoder includes a first training process and a second training process. The first training process includes: Based on the original format sample images and the corresponding label images, the initial diffusion model is trained to obtain the intermediate diffusion model; The second training process includes: The intermediate diffusion model is trained based on the original format sample image and the corresponding label image to obtain the preset diffusion model. During the training of the intermediate diffusion model, the weight parameters of the preset encoder and the preset decoder are adjusted.

12. The image processing method according to claim 11, characterized in that, The training process for the preset diffusion model, preset encoder, and preset decoder also includes: The original format sample image is subjected to neural network super-resolution processing to obtain the labeled image.

13. The image processing method according to claim 11, characterized in that, The adjustment of the weight parameters of the preset encoder and the preset decoder includes: During the training of the intermediate diffusion model, the intermediate sample image and / or output sample image are compared with the label image to obtain difference information. The intermediate sample image is obtained by decoding the intermediate latent vector generated by the intermediate diffusion model through the preset decoder, and the output sample image is obtained by decoding the target latent vector generated by the intermediate diffusion model through the preset decoder. The difference information is used to characterize the degree of difference between the intermediate sample image and / or output sample image and the label image. Based on the difference information, the weight parameters of the preset encoder and the preset decoder are adjusted to reduce the degree of difference.

14. The image processing method according to claim 13, characterized in that, The consistency comparison between the intermediate sample image and / or the output sample image and the label image includes: Perform at least one preset detection on the label image, the intermediate sample image, and / or the output sample image to obtain corresponding detection information; The difference information is determined based on the detection information corresponding to the label image, the intermediate sample image, and / or the output sample image.

15. The image processing method according to claim 14, characterized in that, The preset detection includes at least one of target category detection, text detection, face detection, and texture detection, and the detection information corresponding to the target category detection, text detection, face detection, and texture detection are target category information, text information, face information, and texture information, respectively.

16. The image processing method according to claim 15, characterized in that, The step of adjusting the weight parameters of the preset encoder and the preset decoder based on the difference information includes: In the first training phase, the weight parameters of the preset encoder and the preset decoder are adjusted based on the difference information determined by the target category information. In the second training phase, the weight parameters of the preset encoder and the preset decoder are adjusted based on the difference information determined by the target category information, text information and face information. In the third training phase, the weight parameters of the preset encoder and the preset decoder are adjusted based on the difference information determined by the target category information, text information, face information and texture information.

17. An image processing apparatus, characterized in that, The image processing device includes: A first generation module is used to generate initial latent vectors corresponding to multiple frames of original format images, and the initial latent vectors are used to characterize the image features of the original format images. The second generation module is used to generate a target image corresponding to the multi-frame original format image based on each of the initial latent vectors and the preset diffusion model, so that the imaging quality of the target image is higher than that of the original format image.

18. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image processing method as described in any one of claims 1 to 16.

19. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 16.

Images