Image processing method, image super-resolution model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image processing method, an image super-resolution model training method, an apparatus, an electronic device and a storage medium, and belongs to the field of image processing. The image processing method comprises: performing feature extraction on an original image to obtain first target features, and processing the first target features by using each processing unit in an image super-resolution model to obtain a first feature map and a first classification coefficient corresponding to each processing unit; generating a first weight coefficient corresponding to each processing unit based on the first classification coefficient corresponding to each processing unit; fusing the first feature maps according to the first weight coefficients to obtain a second feature map; and generating a first target image based on the second feature map and the original image, wherein the resolution of the first target image is higher than that of the original image. The image reconstruction capability and efficiency for a content complex image can be improved, the model deployment is simplified, and the degradation effect difference of different information density areas and the model inference time consumption and power consumption are reduced.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, an image super-resolution model training method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] In related technologies, super-resolution algorithms are typically designed to address image / video degradation by using CNN (Convolutional Neural Network) / Transformer methods to fit the data in order to restore the ground truth (real-world annotation).

[0003] For various degradation patterns, a single model struggles to achieve a good fit. For complex degradation patterns or scenarios with high scene complexity, the most common approach is to increase the model's capacity to fit various scenes and degradations, but this increases inference time and power consumption. Furthermore, the different information densities in different regions of the same scene (e.g., flat and textured regions) affect the model's training results. Image super-resolution models perform better in flat regions but worse in dense, high-frequency regions, making it impossible to fit all scenes and difficult to handle the impact of different information density regions within the same scene. Using multiple models for image super-resolution requires adding a classification model, which introduces classification errors. Summary of the Invention

[0004] This disclosure provides an image processing method, an image super-resolution model training method, an apparatus, an electronic device, and a storage medium.

[0005] In a first aspect, this disclosure provides an image processing method, which includes:

[0006] Feature extraction is performed on the original image to obtain the first target feature, and each processing unit in the image super-resolution model is used to process the first target feature to obtain the first feature map and the first classification coefficient corresponding to each processing unit; wherein, the image super-resolution model includes at least two processing units;

[0007] Based on the first classification coefficient corresponding to each processing unit, the first weight coefficient corresponding to each processing unit is generated.

[0008] The first feature maps are fused according to the first weight coefficients to obtain the second feature map;

[0009] A first target image is generated based on the second feature map and the original image, and the resolution of the first target image is higher than that of the original image.

[0010] This embodiment of the image super-resolution model processes the original image separately using each branch (i.e., processing unit) to obtain a first feature map and a first classification coefficient corresponding to each branch. A first weight coefficient is obtained based on the first classification coefficient of each branch. The first weight coefficient and the first feature map are then fused. A high-resolution first target image is generated based on the fused second feature map and the original image. The image super-resolution model of this embodiment is a multi-branch single model, which simplifies model deployment, enables multi-branch collaborative processing, and reduces model inference time and power consumption. During image processing, the weight coefficients are dynamically generated, allowing for adaptive processing of various degradation scenarios and regions with different information densities.

[0011] In some embodiments, a first weight coefficient corresponding to each processing unit is generated based on the first classification coefficient corresponding to each processing unit, including:

[0012] The first classification coefficients corresponding to each processing unit are upsampled to obtain the first processing result;

[0013] The first processing result is normalized to obtain the first weight coefficient corresponding to each processing unit.

[0014] By upsampling the first classification coefficients corresponding to each processing unit, the spatial resolution of the first classification coefficients can be enlarged, and the originally low-resolution first classification coefficients can be accurately mapped onto the high-resolution output image, maintaining consistency with the output resolution.

[0015] In some embodiments, fusing each first feature map according to each first weight coefficient to obtain a second feature map includes:

[0016] The second feature map is obtained by weighting and summing the first weight coefficients and the first feature maps.

[0017] Secondly, this disclosure provides an image super-resolution model training method, which includes:

[0018] Noise is added to the one-hot vectors of at least two image patches of the first sample image to obtain a noisy one-hot vector matrix;

[0019] Based on the noisy one-hot vector matrix and the first sample image, the initial state image super-resolution model is trained to obtain the first training state image super-resolution model; wherein, the image super-resolution model includes at least two processing units, and the number of processing units is the same as the dimension of the one-hot vector matrix.

[0020] Based on the one-hot vector and the first sample image, the image super-resolution model in the first training state is trained a second time to obtain the image super-resolution model in the second training state.

[0021] Each processing unit of the image super-resolution model based on the second training state processes the second sample image to obtain the second classification coefficient and the third feature map corresponding to each processing unit;

[0022] Based on the second sample image, the third weight coefficients corresponding to each processing unit, and each third feature map, the image super-resolution model in the second training state is trained a third time to obtain the trained image super-resolution model; wherein, the third weight coefficients corresponding to each processing unit are obtained based on the second classification coefficients corresponding to the respective processing units.

[0023] This disclosure provides a three-stage progressive training process from noisy guided training to clean supervised training to self-guided training. Noisy guided training ensures stable model initialization and avoids branch training imbalance; clean supervised training strengthens each branch, improving the model's ability to handle specific tasks; and self-guided training frees the model from dependence on external fixed labels, enhancing its adaptability and generalization ability.

[0024] In some embodiments, based on a noisy one-hot vector matrix and a first sample image, an initial-state image super-resolution model is trained to obtain a first-trained image super-resolution model, including:

[0025] The second weighting coefficient is generated based on the noisy one-hot vector matrix;

[0026] Feature extraction is performed on the first sample image to obtain the second target feature. Then, each processing unit in the initial state image super-resolution model is used to process the second target feature to obtain the fourth feature map corresponding to each processing unit.

[0027] The fourth feature maps corresponding to each processing unit are fused according to the second weighting coefficient to obtain the fifth feature map, and the second target image is generated based on the fifth feature map and the first sample image.

[0028] The first loss is determined based on the ground truth annotations corresponding to the second target image and the first sample image, and the first training is performed on the image super-resolution model in the initial state based on the first loss to obtain the image super-resolution model in the first training state.

[0029] In both noisy guided training and clean supervised training, a second weight coefficient based on a one-hot vector matrix is ​​used. This second weight coefficient is accurate, eliminating the need to train the weight allocation and consequently eliminating the need to calculate the loss of the second weight coefficient, thus simplifying the training process.

[0030] In some embodiments, based on the second sample image, each second classification coefficient, and each third feature map, the image super-resolution model in the second training state is subjected to third training to obtain the trained image super-resolution model, including:

[0031] Based on the second classification coefficients corresponding to each processing unit, the third weight coefficients corresponding to each processing unit are generated.

[0032] The third feature maps are fused according to the third weight coefficients to obtain the sixth feature map, and the third target image is generated based on the sixth feature map and the second sample image.

[0033] The image reconstruction loss is determined based on the ground truth annotations corresponding to the third target image and the second sample image. The weight coefficient loss is determined based on the difference between the distribution of the third weight coefficient and the distribution of the target weight coefficient. The third loss is determined based on the image reconstruction loss and the weight coefficient loss.

[0034] The image super-resolution model in the second training state is trained a third time based on the third loss, resulting in the trained image super-resolution model.

[0035] In the third stage of training, the second classification coefficients generated by the image super-resolution model in the second training state are used to generate the third weight coefficients for model training. Therefore, the loss function includes not only the image reconstruction loss, but also the weight coefficient loss calculated based on the third weight. This ensures that the image super-resolution model can achieve a high degree of synergy between "image reconstruction" and "weight allocation" and achieve optimal performance as a whole.

[0036] In some embodiments, before adding noise to the one-hot vectors of at least two image patches of the first sample image, the method further includes:

[0037] The first sample image is divided into at least two image blocks based on a preset size;

[0038] Calculate the evaluation index for each image patch. The evaluation index is used to characterize the image quality of the image patch.

[0039] Based on the evaluation metrics, the image patch is divided into a preset number of image groups to obtain the grouping results of each image patch. The preset number is the number of processing units in the image super-resolution model.

[0040] The one-hot vector of each image block is determined based on the grouping results of each image block.

[0041] Before performing the first training on the initial state of the image super-resolution model, this embodiment first divides and groups the first sample image based on evaluation metrics to achieve automatic data classification and automatically generate a one-hot vector matrix, which serves as the supervision target for the first and second stages of training, enhancing the accuracy of training each branch of the model. By utilizing different branches of the image super-resolution model to focus on processing different types of data and through feature fusion, it achieves better image reconstruction results and higher image reconstruction efficiency than a single model, achieving superior and more adaptive super-resolution performance while controlling model complexity.

[0042] Thirdly, this disclosure provides an image processing apparatus, which includes:

[0043] The feature extraction module is used to extract features from the original image to obtain the first target features;

[0044] At least two processing units, each processing unit is used to process the first target feature to obtain the first feature map and the first classification coefficient corresponding to each processing unit;

[0045] The weight generation module is used to generate the first weight coefficient corresponding to each processing unit based on the first classification coefficient corresponding to each processing unit.

[0046] The feature fusion module is used to fuse the first feature maps according to each first weight coefficient to obtain the second feature map;

[0047] The image super-resolution module is used to generate a first target image based on the second feature map and the original image, wherein the resolution of the first target image is higher than that of the original image.

[0048] Fourthly, this disclosure provides a model training apparatus, which includes:

[0049] The first training module is used to add noise to the one-hot vectors of at least two image patches of the first sample image to obtain a noisy one-hot vector matrix; based on the noisy one-hot vector matrix and the first sample image, the initial state image super-resolution model is trained to obtain the first training state image super-resolution model; wherein, the image super-resolution model includes at least two processing units, and the number of processing units is the same as the dimension of the one-hot vector matrix.

[0050] The second training module is used to perform a second training on the image super-resolution model in the first training state based on the one-hot vector and the first sample image, so as to obtain the image super-resolution model in the second training state.

[0051] The third training module is used to process the second sample image based on each processing unit of the image super-resolution model in the second training state, and obtain the second classification coefficient and third feature map corresponding to each processing unit; based on the second sample image, the third weight coefficient corresponding to each processing unit and each third feature map, the image super-resolution model in the second training state is trained in the third training to obtain the trained image super-resolution model; wherein, the third weight coefficient corresponding to each processing unit is obtained based on the second classification coefficient corresponding to the corresponding processing unit.

[0052] Fifthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the image processing method or image super-resolution model training method described above.

[0053] Sixthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described image processing method or image super-resolution model training method.

[0054] The embodiments provided in this disclosure implement multi-branch processing based on the processing units of the image super-resolution model. The first weight coefficient and first feature map of each branch can be dynamically generated, and adaptive feature fusion is performed on the first weight coefficient and first feature map corresponding to each branch. This can improve the image super-resolution model's ability and efficiency in reconstructing images with complex content. Different branches of the image super-resolution model can adaptively process regions with different characteristics in the original image, reducing the difference in degradation effect in regions with different information density. Multiple types of image degradation problems can be solved using a single image super-resolution model, simplifying model deployment. The collaborative work of multiple lightweight branches can reduce model inference time and power consumption.

[0055] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0056] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0057] Figure 1 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0058] Figure 2 A schematic diagram illustrating image processing using the image super-resolution model provided in this embodiment of the disclosure;

[0059] Figure 3 A flowchart illustrating an image super-resolution model training method provided in this embodiment of the disclosure;

[0060] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0062] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0063] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0064] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0065] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0066] The following defines the terms used in the embodiments of this disclosure:

[0067] The first target feature is the feature obtained by feature extraction from the original image.

[0068] The second target feature is the feature obtained by feature extraction from the first sample image.

[0069] In an image super-resolution model, a processing unit refers to a branch of the image super-resolution model. In this embodiment of the disclosure, the image super-resolution model includes at least two processing units.

[0070] The first feature map can be a feature of some type, such as a frequency, texture, or semantic information.

[0071] The second feature map is the feature map obtained by fusing the processing results of each branch of the image super-resolution model.

[0072] The third feature map is the feature map obtained by each processing unit of the image super-resolution model based on the second training state processing the second sample image.

[0073] The first classification coefficient is used to characterize the matching degree between the image processing capability of the processing unit and the original image.

[0074] The second classification coefficient is obtained by processing the second sample image by each processing unit of the image super-resolution model based on the second training state.

[0075] The first weighting coefficient is used to characterize the contribution weight of the processing unit when processing the original image.

[0076] The second weighting coefficient is a weighting coefficient generated based on a noisy one-hot vector matrix.

[0077] The third weighting coefficient is the weighting coefficient obtained by processing the second classification coefficient.

[0078] The first target image is the final super-resolution image generated through image processing.

[0079] The second target image is generated by using the image super-resolution model of the initial state to generate the super-resolution image corresponding to the first sample image.

[0080] The third target image is an image generated by the image super-resolution model based on the second training state during the third training process.

[0081] To address the problems in related technologies where single models have large capacity and are difficult to handle multiple degradations, while multi-model structures are cumbersome, have classification errors, and exhibit different reconstruction effects for images in regions with different information density, this disclosure provides an image processing method and an image super-resolution model training method based on a multi-branch structure image super-resolution model.

[0082] The image processing method and image super-resolution model training method of the present disclosure embodiments can be executed by electronic devices such as terminal devices or servers. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The image super-resolution model training method can be implemented by the processor calling computer-readable program instructions stored in the memory, or the image super-resolution model training method can be executed by the server.

[0083] Figure 1 A flowchart illustrating an image processing method provided in this disclosure embodiment. Figure 2 This is a schematic diagram illustrating image processing using the image super-resolution model provided in this embodiment of the disclosure. (Refer to...) Figure 1 and Figure 2 The method includes the following steps:

[0084] Step S11: Extract features from the original image to obtain the first target features, and process the first target features using each processing unit in the image super-resolution model to obtain the first feature map and the first classification coefficient corresponding to each processing unit; wherein, the image super-resolution model includes at least two processing units.

[0085] In this step, the feature extraction module can be used to process the original image. Feature extraction is performed to obtain the first target feature. The first target feature is a set of multi-channel feature maps, which includes features from the original image. The feature extraction module extracts various feature information. In some embodiments, the feature extraction module can use an RRDB (Residual Dense Block) structure or a custom structure.

[0086] The image super-resolution model includes at least two processing units, each of which is a branch of the model. In this embodiment, there are k processing units, where k ≥ 2. Each processing unit receives a first target feature as input. Each unit extracts features from the first target feature to obtain a first feature map (feat) corresponding to that unit and generates a first classification coefficient (coe) corresponding to that unit. In other words, the first processing unit outputs its first feature map and first classification coefficient, the second processing unit outputs its first feature map and first classification coefficient, and the kth processing unit outputs its first feature map and first classification coefficient.

[0087] The first feature map can be a feature of some type, such as a certain frequency, texture, or semantic information. The first classification coefficient is used to characterize the image processing capability of the processing unit and the original image. The matching degree. It should be noted that the model parameters of each processing unit are different, which makes different processing units have different abilities to process specific types of data. For example, the convolution kernel weights and biases of two processing units are different. The convolution kernel of one processing unit is good at extracting edges, while the convolution kernel of the other processing unit is good at extracting textures.

[0088] Step S12: Generate the first weight coefficient corresponding to each processing unit based on the first classification coefficient corresponding to each processing unit.

[0089] The first weighting coefficient is used to characterize the processing unit's performance in processing the original image. The contribution weight at time quantifies the proportion of the first feature map of the corresponding branch in the final feature fusion.

[0090] In step S13, the first feature maps are fused according to the first weight coefficients to obtain the second feature map.

[0091] Step S14: Generate a first target image based on the second feature map and the original image. The resolution of the first target image is higher than that of the original image.

[0092] The second feature map is obtained by fusing the processing results of each branch of the image super-resolution model. In this step, residuals can be generated based on the second feature map, and the original image can be processed accordingly. Upsampling is performed based on the residual and the original image. The upsampling results generate a higher resolution second target image. .

[0093] According to embodiments of this disclosure, each processing unit of the image super-resolution model implements multi-branch processing, which can dynamically generate the first weight coefficient and first feature map of each branch. Adaptive feature fusion is performed on the first weight coefficient and first feature map corresponding to each branch, which can improve the image super-resolution model's ability and efficiency in reconstructing images with complex content. Different branches of the image super-resolution model can adaptively process regions with different characteristics in the original image, reducing the difference in degradation effect in regions with different information density. Multiple types of image degradation problems can be solved using a single image super-resolution model, simplifying model deployment. Multiple lightweight branches working together can reduce model inference time and power consumption.

[0094] In some embodiments, generating a first weight coefficient for each processing unit based on a first classification coefficient for each processing unit may include the following steps:

[0095] Step S121: Upsample the first classification coefficients corresponding to each processing unit to obtain the first processing result.

[0096] Step S122: Normalize the first processing result to obtain the first weight coefficient corresponding to each processing unit.

[0097] In some embodiments, the first classification coefficients output by all k processing units are concatenated into a one-dimensional vector, i.e., [coe1, coe2, ..., coek]. This vector is then upsampled to obtain the first processing result. The upsampling process adjusts the dimension of the weight coefficients to a format compatible with subsequent calculations so that they can be input into the Softmax (normalized exponential) layer. The first processing result is then input into the Softmax function (i.e., the normalized exponential function), which exponentializes, sums, and normalizes each element in the vector. The output of the Softmax function is a new vector, denoted as [ , ,..., The elements in this vector are the first weight coefficients corresponding to each processing unit. Each first weight coefficient is between 0 and 1, and the sum of the k first weight coefficients is 1, i.e., 0 ≤ 1. ≤1, and + +...+ =1, i=(1,…,k). The first weight coefficient represents the contribution ratio of the first feature map of each branch to feature fusion when generating the final super-resolution image (i.e., the first target image). The larger the first weight coefficient of the branch, the greater the contribution of its corresponding first feature map to the generation of the first target image.

[0098] By upsampling each of the first classification coefficients, the spatial resolution of the first classification coefficients can be amplified, accurately mapping the originally low-resolution coefficients to the higher-resolution super-resolution image, ensuring consistency with the resolution of the output super-resolution image. Generating the first weight coefficients corresponding to each processing unit lays the foundation for adaptive feature fusion in the image super-resolution model, enabling the model to dynamically determine how to integrate the outputs of each branch based on the input original image.

[0099] In some embodiments, fusing each first feature map according to each first weight coefficient to obtain a second feature map includes the following steps: weighting and summing each first weight coefficient and each first feature map to obtain a second feature map. Specifically, the second feature map can be calculated according to the following formula (1):

[0100] (1)

[0101] in, The first feature map corresponding to the i-th processing unit. Let be the first weight coefficient corresponding to the i-th processing unit, where i is the identifier of the processing unit, i=1,…,k; This is the second feature map.

[0102] By fusing the first feature maps output by each branch using a weighted summation method, the fault tolerance and robustness of the image super-resolution model's decision-making are improved. By integrating the processing results of multiple branches within the image super-resolution model to generate a super-resolution image, the performance, robustness, and efficiency of a single model in handling various complex image degradation scenarios can be improved.

[0103] In some embodiments, generating a first target image based on a second feature map and an original image may include the following steps:

[0104] Step S141: Determine the first residual based on the second feature map.

[0105] In some embodiments, the second feature map can be deep-processed using a residual generation module (conv) to obtain a fourth feature map. The residual generation module is a deep convolutional subnetwork formed by stacking multiple residual blocks, and the fourth feature map is a refined feature map that has been deep-refined and is specifically used for image reconstruction. Through upsampling and image reconstruction, the high-dimensional refined feature map can be mapped to a pixel-level residual image (i.e., the first residual) with the same size as the original image.

[0106] Step S142: Upsample the original image to obtain the second processing result.

[0107] Step S143: Summing the second processing result with the first residual yields the first target image.

[0108] In some embodiments, the first target image can be calculated according to the following formula (2):

[0109] (2)

[0110] in, For upsampling function, This is the upsampling factor. The first target image, For the original image, This is the first residual.

[0111] This disclosure also provides an image super-resolution model training method. Figure 3 A flowchart of an image super-resolution model training method provided in this disclosure embodiment is shown below. Figure 3 As shown, the image super-resolution model training method includes the following steps:

[0112] Step S21: Add noise to the one-hot vectors of at least two image blocks of the first sample image to obtain a noisy one-hot vector matrix.

[0113] In some embodiments, random noise is added to the one-hot vectors of each image block of the first sample data. The noise is uniformly distributed U(0, 0.5) to obtain a noisy one-hot vector matrix. The one-hot vectors are obtained after dividing and grouping the first sample data into image blocks during the data processing stage. The first sample data is divided into at least two image blocks. By grouping the image blocks, a one-hot vector is obtained for each image block. The one-hot vectors of all image blocks form a one-hot vector matrix. By adding random noise to the one-hot vector of each image block, a noisy one-hot vector matrix is ​​obtained.

[0114] Step S22: Based on the noisy one-hot vector matrix and the first sample image, perform the first training on the image super-resolution model in the initial state to obtain the image super-resolution model in the first training state; wherein, the image super-resolution model includes at least two processing units, and the number of processing units is the same as the dimension of the one-hot vector matrix.

[0115] Step S23: Based on the one-hot vector and the first sample image, perform a second training on the image super-resolution model in the first training state to obtain the image super-resolution model in the second training state.

[0116] Step S24: Based on the second training state, each processing unit of the image super-resolution model processes the second sample image to obtain the second classification coefficient and the third feature map corresponding to each processing unit.

[0117] Step S25: Based on the second sample image, the third weight coefficients corresponding to each processing unit, and each third feature map, perform third training on the image super-resolution model in the second training state to obtain the trained image super-resolution model; wherein, the third weight coefficients corresponding to each processing unit are obtained based on the second classification coefficients corresponding to the respective processing units.

[0118] The image super-resolution model training method of this disclosure performs a three-stage progressive training process: "noisy guidance," "clean supervision," and "self-guidance." Steps S21-S22 constitute the first stage, step S23 the second stage, and steps S24-S25 the third stage. In the first stage, the image super-resolution model in its initial state is trained using a one-hot vector matrix with added noise, resulting in a first training state image super-resolution model. This provides stable initialization of the image super-resolution model and avoids branch training imbalance. In the second stage, the image super-resolution model in its first training state is trained using one-hot vectors without added noise, resulting in a second training state image super-resolution model. This strengthens the training of each branch of the image super-resolution model in the second stage, improving the model's ability to process specific types of data. In the third stage, the image super-resolution model in its second training state is self-guided trained based on the second classification coefficients generated from the second training state image super-resolution model, resulting in a trained image super-resolution model. This self-guided training in the third stage decouples the image super-resolution model from its dependence on external fixed labels, enhancing the model's adaptability and generalization ability.

[0119] In some embodiments, based on a noisy one-hot vector matrix and a first sample image, an initial state image super-resolution model is trained to obtain a first training state image super-resolution model, including the following steps:

[0120] Step S221: Generate the second weighting coefficients based on the noisy one-hot vector matrix.

[0121] In this step, the noisy one-hot vector matrix is ​​normalized. For example, the Softmax function can be used for normalization to obtain the second weight coefficient.

[0122] Step S222: Extract features from the first sample image to obtain the second target features, and process the second target features using each processing unit in the initial state image super-resolution model to obtain the fourth feature map corresponding to each processing unit.

[0123] Step S223: The fourth feature map corresponding to each processing unit is fused according to the second weighting coefficient to obtain the fifth feature map, and the second target image is generated based on the fifth feature map and the first sample image.

[0124] In steps S222-S223, the noisy one-hot vector matrix is ​​used as the classification coefficient of each branch of the image super-resolution model in the initial state, and the super-resolution image corresponding to the first sample image, i.e. the second target image, is generated using the image super-resolution model in the initial state.

[0125] Step S224: Determine the first loss based on the real annotations corresponding to the second target image and the first sample image, and perform the first training on the image super-resolution model in the initial state based on the first loss to obtain the image super-resolution model in the first training state.

[0126] In some embodiments, the first loss can be calculated according to the following formula (3):

[0127] (3)

[0128] in, This is the first loss; For the second target image; The first sample image is the ground truth annotation, which is a high-resolution image corresponding to the first sample image that is not degraded and is obtained from the training dataset.

[0129] The process of training an image super-resolution model based on the first loss is a standard deep supervised learning optimization process. Its core is to drive the iterative optimization of the parameters of the multi-branch image super-resolution model in the initial state by minimizing a composite loss function that includes the image fidelity target. Ultimately, the model learns how to generate high-quality super-resolution images and intelligently utilizes its internal multi-branch structure.

[0130] In some embodiments, for example, the gradient of the first loss with respect to each model parameter (including the weights and biases of feature extraction, all branches, Conv modules, etc.) of the initial-state image super-resolution model can be calculated using the backpropagation algorithm. Using an optimizer, such as the Adam (Adaptive Moment Estimation) optimizer or the SGD (Stochastic Gradient Descent) optimizer, all parameters of the initial-state image super-resolution model are updated according to the calculated gradients and a certain learning rate. Training is repeated multiple times on the entire training dataset until the model performance converges or a preset number of training epochs is reached.

[0131] In some embodiments, based on the one-hot vector and the first sample image, the image super-resolution model in the first training state is trained a second time to obtain the image super-resolution model in the second training state, including the following steps:

[0132] Step S231: Generate the fourth weighting coefficient based on the one-hot vector.

[0133] In this step, a one-hot vector matrix is ​​generated based on the one-hot vector, and the one-hot vector matrix is ​​normalized. For example, the Softmax function can be used for normalization to obtain the fourth weight coefficient.

[0134] Step S232: Extract features from the first sample image to obtain the second target features, and process the second target features using each processing unit in the image super-resolution model of the first training state to obtain the seventh feature map corresponding to each processing unit.

[0135] Step S233: The seventh feature map corresponding to each processing unit is fused according to the fourth weighting coefficient to obtain the eighth feature map, and the fourth target image is generated based on the eighth feature map and the first sample image.

[0136] In steps S232-S233, the one-hot vector matrix without added noise is used as the classification coefficients of each branch of the image super-resolution model in the first training state, and the super-resolution image corresponding to the first sample image, i.e. the fourth target image, is generated using the image super-resolution model in the first training state.

[0137] Step S234: Determine the second loss based on the real annotations corresponding to the fourth target image and the first sample image, and perform a second training on the image super-resolution model in the first training state based on the second loss to obtain the image super-resolution model in the second training state.

[0138] It should be noted that the calculation method of the second loss can be the same as that of the first loss, that is, it can be calculated using the above formula (3), except that the second target image in formula (3) is replaced with the fourth target image, and the first loss is replaced with the second loss.

[0139] The process of training the first training state image super-resolution model based on the second loss is a standard deep supervised learning optimization process. Its core is to drive the iterative optimization of the parameters of the multi-branch image super-resolution model in the first training state by minimizing a composite loss function that includes the image fidelity target. Ultimately, the model learns how to generate high-quality super-resolution images and intelligently utilizes its internal multi-branch structure.

[0140] In some embodiments, for example, the gradient of the second loss with respect to each model parameter (including the weights and biases of feature extraction, all branches, Conv modules, etc.) of the image super-resolution model in the first training state can be calculated using the backpropagation algorithm. Using an optimizer (such as Adam or SGD), all parameters of the image super-resolution model in the first training state are updated according to the calculated gradients and a certain learning rate. Training is repeated multiple times on the entire training dataset until the model performance converges or a preset number of training epochs is reached.

[0141] In the first and second training phases, the weight coefficients generated based on the one-hot matrix are used for model training. The one-hot matrix is ​​obtained by dividing and grouping the training sample data, and it is accurate. Therefore, there is no need to train the weight allocation. Accordingly, the loss function can exclude the loss of the weight coefficients, thus simplifying the training process.

[0142] In some embodiments, based on the second sample image, each second classification coefficient, and each third feature map, the image super-resolution model in the second training state is subjected to third training to obtain the trained image super-resolution model, including the following steps:

[0143] Step S251: Generate the third weight coefficient corresponding to each processing unit based on the second classification coefficient corresponding to each processing unit.

[0144] In some embodiments, the second classification coefficients corresponding to each processing unit may be upsampled to obtain a third processing result; the third processing result may be normalized to obtain a third weight coefficient corresponding to each processing unit.

[0145] Step S252: Fuse each third feature map according to each third weight coefficient to obtain a sixth feature map, and generate a third target image based on the sixth feature map and the second sample image.

[0146] In this step, the weighted sum of each third weight coefficient and each third feature map can be obtained to get the sixth feature map; the second residual is determined based on the sixth feature map, the second sample image is upsampled to obtain the third processing result, and the third processing result is summed with the second residual to obtain the third target image. Specifically, the sixth feature map can be calculated using the above formula (1), and the third target image can be calculated using the above formula (2), wherein the original image in formula (2) is replaced with the second sample image, the first residual is replaced with the second residual, and the first target image is replaced with the third target image.

[0147] Step S253: Determine the image reconstruction loss based on the ground truth annotations corresponding to the third target image and the second sample image; determine the weight coefficient loss based on the difference between the distribution of the third weight coefficient and the distribution of the target weight coefficient; and determine the third loss based on the image reconstruction loss and the weight coefficient loss.

[0148] In some embodiments, the third loss can be calculated according to the following formula (4):

[0149] (4)

[0150] in, This is the third loss; For the third target image; The ground truth annotation for the second sample image is a high-resolution image corresponding to the non-degraded second sample image obtained from the training dataset. Image reconstruction loss; Preset grouping coefficients, For the weighted coefficient loss, It can be calculated using the classification cross-entropy loss function or the L1 norm loss function.

[0151] Step S254: Perform third training on the image super-resolution model in the second training state based on the third loss to obtain the trained image super-resolution model.

[0152] The process of training the image super-resolution model in the second training state based on the third loss is a standard deep supervised learning optimization process. Its core is to drive the iterative optimization of the parameters of the multi-branch image super-resolution model in the second training state by minimizing a composite loss function that includes image fidelity target and branch weight supervision target. Ultimately, the model learns how to generate high-quality super-resolution images and intelligently utilizes its internal multi-branch structure.

[0153] In some embodiments, for example, the gradient of the third loss with respect to each model parameter (including the weights and biases of feature extraction, all branches, Conv modules, etc.) of the image super-resolution model in the second training state can be calculated using the backpropagation algorithm. Using an optimizer (such as Adam or SGD), all parameters of the image super-resolution model in the second training state are updated according to the calculated gradients and a certain learning rate. Multiple training rounds are repeated on the entire training dataset until the model performance converges or a preset number of training rounds is reached.

[0154] In the third stage of training, the second classification coefficients generated by the image super-resolution model in the second training state are used to generate the third weight coefficients for model training. Therefore, the loss function includes not only the image reconstruction loss, but also the weight coefficient loss calculated based on the third weight. This ensures that the image super-resolution model can achieve a high degree of synergy between "image reconstruction" and "weight allocation" and achieve optimal performance as a whole.

[0155] In some embodiments, before adding noise to the one-hot vectors of at least two image patches of the first sample image, the image super-resolution model training method may further include the following steps:

[0156] Step S31: Divide the first sample image into at least two image blocks based on a preset size.

[0157] In this step, the first sample image is divided into multiple image blocks of the same size according to a preset size.

[0158] Step S32: Calculate the evaluation index for each image patch. The evaluation index is used to characterize the image quality of the image patch.

[0159] In some embodiments, for each image patch, image reconstruction can be performed on the image patch to generate a target image patch for the image patch; based on the target image patch and the corresponding ground truth annotation, the evaluation index of the image patch is calculated, wherein the resolution of the target image patch is higher than the resolution of the image patch.

[0160] In some embodiments, the evaluation metrics may include, but are not limited to, PSNR (Peak Signal-to-Noise Ratio) and L1 norm loss function.

[0161] Step S33: Based on the evaluation index, the image block is divided into a preset number of image groups to obtain the grouping results of each image block. The preset number is the number of processing units in the image super-resolution model.

[0162] Step S34: Determine the one-hot vector of each image block based on the grouping results of each image block.

[0163] For an image patch belonging to group j, the j-th element in its unique vector is set to 1, while the other elements are set to 0. This is to encourage the j-th branch of the image super-resolution model to play a dominant role, by setting the feature weight of this branch to 1 and the feature weights of the other branches to 0. For example, for an image patch belonging to group 2, its one-hot vector can be represented as [0, 1, 0, ..., 0].

[0164] It should be noted that steps S31-S34 above can be performed using an image processing model, which can be a State of the Art (SOTA) model or a custom model.

[0165] Before performing the first training on the initial state of the image super-resolution model, this embodiment first divides and groups the first sample image based on evaluation metrics to achieve automatic data classification and automatically generate a one-hot vector matrix, which serves as the supervision target for the first and second stages of training, enhancing the accuracy of training each branch of the model. By utilizing different branches of the image super-resolution model to focus on processing different types of data and through feature fusion, it achieves better image reconstruction results and higher image reconstruction efficiency than a single model, achieving superior and more adaptive super-resolution performance while controlling model complexity.

[0166] The embodiments disclosed herein employ a single model to achieve image super-resolution processing, reducing the risk of error propagation caused by multiple independent classifiers. The model has branch adaptive capability, which can better cope with the huge differences between different regions within a single image, resulting in more consistent and reliable output results and stronger robustness. This improvement in robustness makes the embodiments disclosed herein more stable in the face of diverse and unpredictable image inputs.

[0167] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0168] In addition, this disclosure also provides an image processing apparatus, a model training apparatus, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the image processing methods or image super-resolution model training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0169] This disclosure also provides an image processing apparatus, which includes:

[0170] The feature extraction module is used to extract features from the original image to obtain the first target features.

[0171] At least two processing units, each processing unit is used to process the first target feature to obtain the first feature map and the first classification coefficient corresponding to each processing unit.

[0172] The weight generation module is used to generate the first weight coefficient for each processing unit based on the first classification coefficient corresponding to each processing unit.

[0173] The feature fusion module is used to fuse the first feature maps according to the first weight coefficients to obtain the second feature map.

[0174] The image super-resolution module is used to generate a first target image based on the second feature map and the original image, wherein the resolution of the first target image is higher than that of the original image.

[0175] In some embodiments, the weight generation module is used to perform upsampling processing on the first classification coefficients corresponding to each processing unit to obtain a first processing result; and to perform normalization processing on the first processing result to obtain the first weight coefficients corresponding to each processing unit.

[0176] In some embodiments, the feature fusion module is used to perform a weighted summation of each first weight coefficient and each first feature map to obtain a second feature map.

[0177] This disclosure also provides a model training apparatus, which includes:

[0178] The first training module is used to add noise to the one-hot vectors of at least two image patches of the first sample image to obtain a noisy one-hot vector matrix; based on the noisy one-hot vector matrix and the first sample image, the initial state image super-resolution model is trained to obtain the first training state image super-resolution model; wherein the image super-resolution model includes at least two processing units, and the number of processing units is the same as the dimension of the one-hot vector matrix.

[0179] The second training module is used to perform a second training on the image super-resolution model in the first training state based on the one-hot vector and the first sample image, so as to obtain the image super-resolution model in the second training state.

[0180] The third training module is used to process the second sample image based on each processing unit of the image super-resolution model in the second training state, and obtain the second classification coefficient and third feature map corresponding to each processing unit; based on the second sample image, the third weight coefficient corresponding to each processing unit and each third feature map, the image super-resolution model in the second training state is trained in the third training to obtain the trained image super-resolution model; wherein, the third weight coefficient corresponding to each processing unit is obtained based on the second classification coefficient corresponding to the corresponding processing unit.

[0181] In some embodiments, the first training module is used to: generate second weight coefficients based on a noisy one-hot vector matrix; extract features from a first sample image to obtain second target features, and process the second target features using each processing unit in the initial state image super-resolution model to obtain a fourth feature map corresponding to each processing unit; fuse the fourth feature maps corresponding to each processing unit according to the second weight coefficients to obtain a fifth feature map, and generate a second target image based on the fifth feature map and the first sample image; determine a first loss based on the ground truth annotations corresponding to the second target image and the first sample image, and perform a first training on the initial state image super-resolution model based on the first loss to obtain an image super-resolution model in a first training state.

[0182] In some embodiments, the third training module is used to: generate third weight coefficients corresponding to each processing unit based on the second classification coefficients corresponding to each processing unit; fuse each third feature map according to each third weight coefficient to obtain a sixth feature map, and generate a third target image based on the sixth feature map and the second sample image; determine the image reconstruction loss based on the ground truth annotations corresponding to the third target image and the second sample image, determine the weight coefficient loss based on the difference between the distribution of the third weight coefficients and the distribution of the target weight coefficients, and determine the third loss based on the image reconstruction loss and the weight coefficient loss; and perform third training on the image super-resolution model in the second training state based on the third loss to obtain the trained image super-resolution model.

[0183] In some embodiments, the model training apparatus further includes a data processing module, which is configured to: divide a first sample image into at least two image blocks based on a preset size; calculate an evaluation index for each image block, the evaluation index being used to characterize the image quality of the image block; divide the image block into a preset number of image groups based on the evaluation index, obtaining the grouping result of each image block, the preset number being the number of processing units in the image super-resolution model; and determine the one-hot vector of each image block based on the grouping result of each image block.

[0184] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0185] Reference Figure 4 This disclosure provides an electronic device, which includes: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by the at least one processor 401, and the one or more computer programs are executed by the at least one processor 401 to enable the at least one processor 401 to perform the above-described image processing method or image super-resolution model training method.

[0186] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned image processing method or image super-resolution model training method. The computer-readable storage medium may be volatile or non-volatile.

[0187] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0188] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0189] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0190] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0191] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0192] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0193] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0194] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0196] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image processing method, characterized in that, include: Feature extraction is performed on the original image to obtain a first target feature, and each processing unit in the image super-resolution model is used to process the first target feature to obtain a first feature map and a first classification coefficient corresponding to each processing unit; wherein, the image super-resolution model includes at least two processing units; Based on the first classification coefficient corresponding to each processing unit, a first weight coefficient corresponding to each processing unit is generated. The first feature maps are fused according to the first weight coefficients to obtain the second feature map; A first target image is generated based on the second feature map and the original image, wherein the resolution of the first target image is higher than that of the original image.

2. The method according to claim 1, characterized in that, The step of generating a first weight coefficient for each processing unit based on the first classification coefficient for each processing unit includes: The first classification coefficients corresponding to each processing unit are upsampled to obtain the first processing result; The first processing result is normalized to obtain the first weight coefficient corresponding to each processing unit.

3. The method according to claim 1, characterized in that, The step of fusing each of the first feature maps according to each of the first weight coefficients to obtain a second feature map includes: The second feature map is obtained by weighted summation of each of the first weight coefficients and each of the first feature maps.

4. A method for training an image super-resolution model, characterized in that, include: Noise is added to the one-hot vectors of at least two image patches of the first sample image to obtain a noisy one-hot vector matrix; Based on the noisy one-hot vector matrix and the first sample image, the initial state image super-resolution model is trained to obtain the first training state image super-resolution model; wherein, the image super-resolution model includes at least two processing units, and the number of processing units is the same as the dimension of the one-hot vector matrix; Based on the one-hot vectors of the at least two image patches and the first sample image, the image super-resolution model in the first training state is trained a second time to obtain the image super-resolution model in the second training state. Each processing unit of the image super-resolution model based on the second training state processes the second sample image to obtain the second classification coefficient and the third feature map corresponding to each processing unit; Based on the second sample image, the third weight coefficients corresponding to each processing unit, and each third feature map, the image super-resolution model in the second training state is subjected to third training to obtain the trained image super-resolution model; wherein, the third weight coefficients corresponding to each processing unit are obtained based on the second classification coefficients corresponding to the respective processing unit.

5. The method according to claim 4, characterized in that, The first training of the image super-resolution model in the initial state, based on the noisy one-hot vector matrix and the first sample image, to obtain the image super-resolution model in the first training state includes: The second weighting coefficient is generated based on the noisy one-hot vector matrix; Feature extraction is performed on the first sample image to obtain the second target feature, and each processing unit in the image super-resolution model of the initial state is used to process the second target feature to obtain the fourth feature map corresponding to each processing unit; The fourth feature maps corresponding to each processing unit are fused according to the second weighting coefficient to obtain a fifth feature map, and a second target image is generated based on the fifth feature map and the first sample image. The first loss is determined based on the ground truth annotations corresponding to the second target image and the first sample image, and the first training is performed on the image super-resolution model in the initial state based on the first loss to obtain the image super-resolution model in the first training state.

6. The method according to claim 4, characterized in that, The step of performing a third training on the image super-resolution model in the second training state based on the second sample image, each of the second classification coefficients, and each of the third feature maps to obtain the trained image super-resolution model includes: Based on the second classification coefficient corresponding to each processing unit, a third weight coefficient corresponding to each processing unit is generated. The third feature maps are fused according to the third weight coefficients to obtain the sixth feature map, and the third target image is generated based on the sixth feature map and the second sample image. The image reconstruction loss is determined based on the ground truth annotations corresponding to the third target image and the second sample image. The weight coefficient loss is determined based on the difference between the distribution of the third weight coefficient and the distribution of the target weight coefficient. The third loss is determined based on the image reconstruction loss and the weight coefficient loss. The image super-resolution model in the second training state is trained a third time based on the third loss to obtain the trained image super-resolution model.

7. The method according to any one of claims 4-6, characterized in that, Before adding noise to the one-hot vectors of at least two image patches of the first sample image, the method further includes: The first sample image is divided into at least two image blocks based on a preset size; Calculate the evaluation index for each of the image blocks, the evaluation index being used to characterize the image quality of the image blocks; Based on the evaluation index, the image block is divided into a preset number of image groups to obtain the grouping results of each image block. The preset number is the number of processing units in the image super-resolution model. The one-hot vector of each image block is determined based on the grouping results of each image block.

8. An image processing apparatus, characterized in that, include: The feature extraction module is used to extract features from the original image to obtain the first target features; At least two processing units, each processing unit is used to process the first target feature to obtain a first feature map and a first classification coefficient corresponding to each processing unit; The weight generation module is used to generate a first weight coefficient for each processing unit based on the first classification coefficient corresponding to each processing unit. The feature fusion module is used to fuse each of the first feature maps according to each of the first weight coefficients to obtain a second feature map; An image super-resolution module is used to generate a first target image based on the second feature map and the original image, wherein the resolution of the first target image is higher than the resolution of the original image.

9. A model training device, characterized in that, include: The first training module is used to add noise to the one-hot vectors of at least two image patches of the first sample image to obtain a noisy one-hot vector matrix. Based on the noisy one-hot vector matrix and the first sample image, the initial state image super-resolution model is trained to obtain the first training state image super-resolution model; wherein, the image super-resolution model includes at least two processing units, and the number of processing units is the same as the dimension of the one-hot vector matrix; The second training module is used to perform a second training on the image super-resolution model in the first training state based on the one-hot vectors of the at least two image blocks and the first sample image, so as to obtain the image super-resolution model in the second training state. The third training module is used to process the second sample image based on each processing unit of the image super-resolution model in the second training state, to obtain the second classification coefficient and the third feature map corresponding to each processing unit; and to perform a third training on the image super-resolution model in the second training state based on the second sample image, the third weight coefficient corresponding to each processing unit, and the third feature map, to obtain the trained image super-resolution model; wherein the third weight coefficient corresponding to each processing unit is obtained based on the second classification coefficient corresponding to the corresponding processing unit.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the image processing method as described in any one of claims 1-3, or the image super-resolution model training method as described in any one of claims 4-7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method as described in any one of claims 1-3, or the image super-resolution model training method as described in any one of claims 4-7.

Images