Image processing method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides an image processing method and device, equipment, a storage medium and a product, and the method comprises the steps: extracting the image features of a to-be-processed image, and obtaining a feature map; sorting the feature elements included in the feature map according to the size relationship of the feature elements included in the feature map to obtain a sorted feature map; performing first feature recombination processing on the sorted feature map according to a preset number of feature element categories to obtain a first recombined feature map, and performing second feature recombination processing on the sorted feature map according to a preset number of elements under each feature element category to obtain a second recombined feature map; and generating an attention map according to the first recombined feature map and the second recombined feature map, and performing reconstruction processing on the to-be-processed image according to the attention map to obtain a recovered image. According to the technical scheme provided by the embodiment of the invention, the pixels with the similar pixel values obtained through sorting can be classified, so that clear information and degraded characteristics can be focused, and the image restoration effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to the field of computer vision, and more particularly to an image processing method and device, equipment, storage medium and product. BACKGROUND

[0002] With the development of electronic devices and computer technology, users' requirements for image quality are gradually increasing. Therefore, image restoration technology for reconstructing or restoring images with degraded quality has become one of the most important image processing technologies in the field of computer vision. Recovering images affected by adverse weather conditions is an important research topic in image restoration technology. Adverse weather conditions can include fog, rain, or snow, etc. Adverse weather conditions usually result in poor visual quality of the captured images, which can result in reduced clarity, contrast, and color saturation of the images, and increased noise and blur of the images. These images taken under adverse weather conditions not only affect people's daily travel, but also affect the performance of downstream tasks, such as autonomous driving systems, target detection tasks, depth detection tasks, etc. Therefore, how to recover clear and real images from images taken under adverse weather conditions is a technical problem that needs to be solved at present.

[0003] Currently, weather-related prior knowledge can be used to model the statistical properties of degradation factors to eliminate the visual effects of adverse weather. However, the recovered images obtained in this way still contain some effects of adverse weather, and the image restoration effect is poor, which cannot meet the demand for image quality in actual application scenarios. SUMMARY

[0004] The embodiments of the present application provide an image processing method, device, equipment, storage medium and product, which can improve the image restoration effect of degraded images affected by weather.

[0005] In a first aspect, the embodiments of the present application provide an image processing method, comprising:

[0006] extracting image features of a to-be-processed image to obtain a feature map corresponding to the to-be-processed image;

[0007] performing sorting processing on feature elements included in the feature map according to a size relationship of the feature elements, to obtain a sorted feature map;

[0008] performing first feature reorganization processing on the sorted feature map according to a preset number of feature element categories to obtain a first reorganized feature map, and performing second feature reorganization processing on the sorted feature map according to a preset number of elements under each feature element category to obtain a second reorganized feature map;

[0009] The generation unit is configured to generate an attention map according to the first reorganized feature map and the second reorganized feature map, and perform reconstruction processing on the to-be-processed image according to the attention map, to obtain a recovery image corresponding to the to-be-processed image.

[0010] In a second aspect, an image processing apparatus is provided, including:

[0011] The extraction unit is configured to extract image features of a to-be-processed image, to obtain a feature map corresponding to the to-be-processed image.

[0012] The sorting unit is configured to perform sorting processing on feature elements included in the feature map according to a size relationship of the feature elements, to obtain a sorted feature map.

[0013] The processing unit is configured to perform first feature reorganization processing on the sorted feature map according to a preset number of feature element categories, to obtain a first reorganized feature map, and perform second feature reorganization processing on the sorted feature map according to a preset number of elements under each feature element category, to obtain a second reorganized feature map.

[0014] The generation unit is configured to generate an attention map according to the first reorganized feature map and the second reorganized feature map, and perform reconstruction processing on the to-be-processed image according to the attention map, to obtain a recovery image corresponding to the to-be-processed image.

[0015] In a third aspect, an image processing device is provided, including one or more processors, and a memory configured to store one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method of the first aspect.

[0016] In a fourth aspect, a computer readable storage medium is provided, which stores instructions, which, when executed on a computer, cause the computer to execute the image processing method of the first aspect.

[0017] In a fifth aspect, a computer program product is provided, which includes computer programs or computer instructions, which, when executed by a processor, implement the image processing method of the first aspect.

[0018] In the technical solutions provided in some embodiments of the present application, by extracting image features of a to-be-processed image, a feature map of a degraded image is obtained, then the feature elements included in the feature map are sorted according to the size relationship of the feature elements, to obtain a sorted feature map, then the sorted feature map is subjected to first feature reorganization processing according to a preset number of feature element categories, to obtain a first reorganized feature map, and the sorted feature map is subjected to second feature reorganization processing according to a preset number of elements in each feature element category, to obtain a second reorganized feature map; then an attention map is generated according to the first reorganized feature map and the second reorganized feature map, and the to-be-processed image is reconstructed according to the attention map, to obtain a recovered image corresponding to the to-be-processed image. As can be seen, since the pixels affected by the weather are usually close in pixel value, the sorting processing can determine the pixels close in pixel value and dispersed in space, so that the clear pixels and the affected pixels are separated, which is conducive to extracting clear image features and degraded image features. By classifying the sorted pixels based on two different ways respectively, clear features and degraded features of different dimensions can be extracted respectively, an attention map focusing on preserving clear features and recovering degraded features can be generated, so that the image recovery effect can be improved to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a schematic diagram of an image processing system architecture provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a to-be-processed image affected by different bad weather provided by an embodiment of the present application;

[0023] Figure 4 is a structural schematic diagram of an image processing model provided by an embodiment of the present application;

[0024] Figure 5 is a structural schematic diagram of an image processing module provided by an embodiment of the present application;

[0025] Figure 6 is a principle schematic diagram of feature element classification provided by an embodiment of the present application;

[0026] Figure 7 A principle diagram of a first recombination processing provided by an embodiment of the present application;

[0027] Figure 8 A principle diagram of a second recombination processing provided by an embodiment of the present application;

[0028] Figure 9 A processing timing diagram of a DHSA module provided by an embodiment of the present application;

[0029] Figure 10 A principle diagram of a DGFF module provided by an embodiment of the present application;

[0030] Figure 11 Another flow diagram of an image processing method provided by an embodiment of the present application;

[0031] Figure 12 An evaluation effect diagram of an image processing method provided by an embodiment of the present application;

[0032] Figure 13 A comparison effect diagram of a recovered image provided by an embodiment of the present application;

[0033] Figure 14 Another comparison effect diagram of a recovered image provided by an embodiment of the present application;

[0034] Figure 15 An effect comparison diagram of performing a downstream task provided by an embodiment of the present application;

[0035] Figure 16 Another effect comparison diagram of performing a downstream task provided by an embodiment of the present application;

[0036] Figure 17 A structure diagram of another image processing apparatus provided by an embodiment of the present application;

[0037] Figure 18 A structure diagram of an image processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] It should be noted that, in order for the person skilled in the art to better understand the technical solutions put forward by the embodiments of the present application, the embodiments of the present application will combine one or more drawings to clearly and completely describe the implementation manner of the technical solutions put forward by the embodiments of the present application. And, the various drawings shown in the embodiments of the present application are only exemplary descriptions, for example, the execution order of each step in the drawings can be adaptively adjusted according to the actual application scene. In addition, in the embodiments of the present application, the block diagram shown in each drawing is only a functional entity, which does not necessarily correspond to a physically independent entity. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuit or memory) or combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that contains the functions of the module or unit.

[0040] It should be noted that "multiple" mentioned in this paper refers to two or more than two. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are a "or" relationship.

[0041] In the process of image restoration processing of images affected by adverse weather conditions, it is found that in order to improve the visual quality of images affected by degradation factors in weather conditions, image restoration processing can be performed based on statistical characteristics of degradation factors, or image restoration processing can be performed through a machine learning (ML) model. Among them, the image restoration processing based on the statistical characteristics of the degradation factors refers to modeling the statistical characteristics of the degradation factors based on weather-related prior knowledge, which is used to eliminate the visual impact of adverse weather. The effect of this way of image restoration is poor, and the restored image contains the influence of part of the adverse weather, and the visual quality of the image is poor. The image restoration processing through the ML model refers to training a convolutional neural network (CNN) to solve the problems of rain removal, defogging and snow removal, etc. The performance of this way is better than that of the image restoration processing based on statistical characteristics, but it needs to train the network for each task separately, that is, different adverse weather (such as fog, rain and snow, etc.) needs to train the network separately to meet the needs of practical application. Although a pre-trained model (i.e. all-in-one model) for completing multiple tasks can be trained to perform the adverse weather removal task, and a single model is used to restore images affected by different weather conditions. But through experiments, it is found that the performance of this way of model is also relatively poor, and the restored image cannot meet the improvement of the aesthetic clarity of the human eye vision, and may also affect the performance of the downstream task. Therefore, image restoration of degraded images affected by adverse weather is crucial for the aesthetic clarity of human visual observation and the performance of visual tasks.

[0042] Based on this, the embodiment of the present application provides an image processing method, which can be used to perform the task of all-in-one adverse weather removal, and perform image restoration processing on degraded images affected by adverse weather conditions such as fog, rain and snow, eliminate low-quality (degradation) information in the degraded images, and retain clear information in the degraded images, so as to improve the image restoration effect of the degraded images affected by weather to a certain extent, improve the performance of image restoration, and also help to improve the performance of downstream tasks.

[0043] The image processing scheme provided by the embodiment of the present application relates to artificial intelligence, machine learning, computer vision and other technologies.

[0044] Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation (interaction) system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning (deep learning) and other major directions.

[0045] ML is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning (Deep Learning, DL) usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0046] Computer vision (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further process images to make computer processing more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, and tries to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0047] Based on the above description, the image processing scheme provided in the embodiments of the present application can improve the image recovery effect. Please refer to Figure 1 , Figure 1 is a schematic diagram of an architecture of an image processing system provided in the embodiments of the present application, as shown in Figure 1 , the image processing system includes an image processing device 101, an image acquisition device 102, and a downstream task device 103, and the image processing scheme provided in the present application can be executed by the model processing device 101. Wherein, the image processing device 101 can be directly or indirectly connected with the image acquisition device 102 through wired or wireless mode. It should be noted that Figure 1 the number and form of the devices shown are used for example, and do not constitute a limitation on the embodiments of the present application. Alternatively, the image processing device 101 and the image acquisition device 102 can be the same electronic device, or can be two different electronic devices. Alternatively, the image processing device 101 can also be the same device as the image acquisition device 102 and the downstream task device 103, or can be different electronic devices, which are not limited in the present application. The embodiments of the present application take the image processing device 101, the image acquisition device 102 and the downstream task device 103 as the same electronic device for example.

[0048] Among them, the image processing device 101, the image acquisition device 102 and the downstream task device 103 can be terminal devices or servers, which can include but not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, MID), smart voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, wearable devices, etc., which are not limited in the embodiments of the present application; the server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platform, which are not limited in the embodiments of the present application.

[0049] The general process of the image processing scheme provided in the present application is as follows:

[0050] The image processing device 101 can extract image features of the to-be-processed image to obtain a feature map corresponding to the to-be-processed image. After obtaining the feature map, the image processing device 101 can perform sorting processing on feature elements included in the feature map according to a size relationship of the feature elements, to obtain a sorted feature map. Further, the image processing device 101 can perform first feature reorganization processing on the sorted feature map according to a preset number of feature element categories, to obtain a first reorganized feature map, and perform second feature reorganization processing on the sorted feature map according to a preset number of elements under each feature element category, to obtain a second reorganized feature map. Finally, the image processing device 101 can generate an attention map according to the first reorganized feature map and the second reorganized feature map, and perform reconstruction processing on the to-be-processed image according to the attention map, to obtain a restored image corresponding to the to-be-processed image.

[0051] The to-be-processed image can be a degraded image affected by a severe weather condition, which is captured by the image acquisition device 102 through a capturing device, generated by the image acquisition device 102 through an image generation tool (such as a text-to-image model), or obtained by the image acquisition device 102 from the Internet. The application does not limit the to-be-processed image. After obtaining the restored image corresponding to the to-be-processed image, the restored image can be used for downstream task device 103 to perform downstream tasks, such as target detection tasks for recognizing objects contained in the image, depth estimation tasks for estimating the distance between objects in the image and the camera, and the like.

[0052] In the case that the to-be-processed image is a training image in a training sample, the training sample can include a clean image as a reference, i.e., a label image. The image processing device 101 can train an initial image processing model based on the training sample to obtain a target image processing model. Specifically, the image processing device 101 can determine first loss data according to the difference between the obtained restored image and the label image, and determine second loss data according to the correlation between image blocks in the restored image and image blocks in the label image. Further, the image processing device 101 can determine target loss data according to the sum of the first loss data and the second loss data. Finally, the image processing device 101 trains the initial image processing model based on the target loss data to obtain the target image processing model. It can be understood that the target image processing model can be used for image restoration of degraded images affected by different severe weather conditions (such as fog, rain, or snow, etc.), to improve the image quality of the degraded images.

[0053] Since pixels affected by weather are usually similar in pixel value, based on the above image processing system, by performing sorting processing, pixels similar in pixel value but with uncertain (more dispersed) spatial positions can be obtained, so that clear pixels and affected pixels are separated. Further, by classifying the sorted pixels based on two different ways, clear features and degraded features of different dimensions can be extracted respectively, an attention map focusing on preserving clear features and recovering degraded features can be generated, thereby improving the image recovery effect to a certain extent.

[0054] In an implementation manner, the to-be-processed image and the recovered image can be stored in a blockchain, so that the to-be-processed image and the recovered image can be prevented from being tampered with. The blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies, and its essence is a decentralized database, that is, a series of data blocks associated using cryptographic methods, each data block contains information of a batch of network transactions, and is used to verify the validity (anti-fake) of the information and generate the next block.

[0055] It can be understood that the image processing system described in the embodiments of the present application is for more clearly illustrating the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be known by those skilled in the art that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0056] Based on the above image processing system, the embodiments of the present application provide an image processing method. The image processing method described in the embodiments of the present application can be executed by an electronic device, which can be an image processing device 101 in the image processing system shown in Figure 1 . Please refer to Figure 2 , Figure 2 is a flowchart of an image processing method provided by the embodiments of the present application. The image processing method includes the following steps S201-S204:

[0057] S201, extracting image features of a to-be-processed image to obtain a feature map corresponding to the to-be-processed image.

[0058] In the embodiments of the present application, the to-be-processed image is an image affected by severe weather conditions, which can also be referred to as a degraded image. The severe weather conditions can be rain, fog, snow, etc. Please refer to Figure 3 , Figure 3 is a schematic diagram of a to-be-processed image affected by different adverse weather, wherein Figure 3 (a) in FIG. 1 indicates a to-be-processed image affected by rain and fog, Figure 3 (b) in FIG. 1 indicates a to-be-processed image affected by raindrops,Figure 3 The middle (c) represents the to-be-processed image affected by snow. The to-be-processed image can be obtained from the image in the middle (a)- Figure 3 The middle (c) represents the to-be-processed image affected by snow. The to-be-processed image can be obtained from the image in the middle (a)- Figure 3 As seen in the middle (c), different degradation factors caused by adverse weather generally show similar patterns, that is, the image may contain specific textures, brightness changes or color shifts due to weather conditions such as fog, rain, snow, etc., which visually present consistent or repetitive features or structures.

[0059] In a possible implementation, the image processing device can extract image features of the to-be-processed image through a deep learning model, such as an image processing model, to obtain a feature map corresponding to the to-be-processed image. The feature map is a data structure commonly used in deep learning, which is obtained by convolution processing of the to-be-processed image. Each pixel in the feature map can represent a specific feature of the captured to-be-processed image, and the captured feature can be used for subsequent completion of a specific task, such as image restoration processing. It can be understood that the feature map can be regarded as an abstract representation of the original image (to-be-processed image), which is the result of the intermediate layer in the image processing model and can be further passed to the subsequent network (layer) for processing.

[0060] It should be noted that the feature map (such as the above feature map, also as the initial feature map in the following, etc.) generally has a three-dimensional structure and is a three-dimensional tensor. The essence of a tensor is a multidimensional array, which can represent data in different dimensions. For example, a grayscale image can be regarded as a two-dimensional tensor (or matrix), and each feature element (which can also be referred to as an element) in the two-dimensional tensor (matrix) represents the grayscale value (brightness value) of a pixel. For another example, a color image can be regarded as a three-dimensional tensor, and each pixel has three channels, respectively representing the intensity of red, green and blue color channels. The above to-be-processed image can be represented as I lq ∈R 3×H×W , I lq represents a low-quality image, and the expression can be used to represent I lq is a three-dimensional tensor, and its elements belong to a real number set R, and the shape of the tensor is 3xHxW, 3 represents color channels such as red, green and blue, and H and W represent the height and width of the to-be-processed image.

[0061] wherein, for a feature map, the first dimension of the three-dimensional structure represents the number of channels (C), which can represent the number of channel feature maps included in the feature map. Since, in a certain layer of the model, the convolution kernel can act on the input image (such as the to-be-processed image) or the feature map of the previous layer, each convolution kernel will generate a corresponding channel feature map, and therefore C also represents the number of channel feature maps generated by this layer. In the embodiments of the present application, the set of channel feature maps generated by a certain layer or a certain network is referred to as a feature map. The other two dimensions represent the height (H) and the width (W), which can form a two-dimensional matrix to describe the spatial size of the feature map. Each position (each pixel) of the feature map usually corresponds to a feature representation of a certain local region of the input image (such as the to-be-processed image) or the feature map of the previous layer.

[0062] Next, the process of extracting image features of the to-be-processed image will be described in detail in combination with the model structure of the image processing model provided by the embodiments of the present application.

[0063] Please refer to Figure 4 , Figure 4 is a structural diagram of an image processing model provided by the embodiments of the present application, as shown in Figure 4 , the image processing model is a transformer model, which is used to remove the influence of various bad weather in the image. The transformer model can be referred to as a Histoformer. The network backbone of the image processing model includes an encoder and a decoder. The processing part of the encoder can be understood as an extraction process of clear features and degraded features, and the processing part of the decoder can be understood as a reconstruction and recovery process of the to-be-processed image. Wherein, the encoder and the decoder can each include a plurality of blocks, i.e. the image processing module shown in Figure 4 , which can also be referred to as a Histoformer module, is a key component of the image processing model, which is used to extract complex features and capture dynamic distribution of degradation factors. For the convenience of description, the embodiments of the present application take the encoder and the decoder each including 4 image processing modules as an example for explanation, and the number of image processing modules included in the encoder and the decoder is not limited by the present application.

[0064] Specifically, the encoder can be composed of Figure 4 four image processing modules on the left, which can also be referred to as encoder layers, i.e. the encoder is stacked by four encoder layers, such as L1, L2, L3 and L4 from top to bottom. Similarly, the decoder can be composed of Figure 4The four image processing modules on the right side form the decoder layer, that is, the decoder is stacked by four decoder layers, such as L1, L2, L3 and L4 from bottom to top. The processing of each image processing module can be called a stage. In the encoder, between adjacent two stages, the feature map output by the previous stage can be down-sampled to double the number of channels, and the width and height of the feature map size are halved, so that the down-sampled feature map can be used as the input of the next stage, as shown by the dashed lower arrow between adjacent image processing modules in FIG. 2. Figure 4 In the decoder, between adjacent two stages, the feature map output by the previous stage can be up-sampled to halve the number of channels, and the width and height of the feature map size are doubled, so that the up-sampled feature map can be used as the input of the next stage, as shown by the dashed upper arrow between adjacent image processing modules in FIG. 2. Figure 4

[0065] Optionally, the down-sampling processing involved in the above-mentioned encoder can be realized by pixel restoration / pixel inverse reorganization (Pixel Unshuffle) technology, the core idea of which is to combine the pixels in a high-resolution feature map into one pixel in a low-resolution feature map. This is usually achieved by aggregating (such as averaging, maximum, etc.) multiple pixel values in a high-resolution feature map.

[0066] Optionally, the up-sampling processing involved in the above-mentioned decoder can be realized by pixel shuffling / pixel reorganization (Pixel Shuffle) technology, which is to "expand" each pixel value in a low-resolution feature map to a region in a high-resolution feature map. It is usually achieved by rearranging the pixels in the feature map to generate an output image (feature map) with higher resolution. For example, the shape of a low-resolution feature map is r 2 ×H×W, r 2 is the number of channels, and a new high-resolution feature Figure 1 ×rH×rW is formed by rearranging each pixel.

[0067] Optionally, the down-sampling processing involved in the above-mentioned encoder can also be realized by pooling processing, and the up-sampling processing involved in the above-mentioned decoder can also be realized by different interpolation algorithms to estimate the value of the new pixel, thereby increasing the size of the image. The present application does not limit this, and the specific model structure and processing process can be determined according to the actual task scenario.

[0068] Among them, for the same size stage, it can be called the same stage, and in the same stage, the encoder and the decoder can be connected to each other through a skip connection, as shown in FIG. 2. Figure 4 ​The skip connections between the image processing modules in the encoder and their corresponding modules in the decoder establish connections between consecutive intermediate features, allowing gradients to propagate more easily during training. This helps mitigate the vanishing or exploding gradient problem, thus enhancing training stability. Furthermore, in the encoder and decoder structure, skip connections enable direct information flow between different layers, helping the network learn richer feature representations. Additionally, the image processing modules are preceded and followed by 1×1 convolutional processing. This 1×1 indicates a 1×1 kernel size, which can be used to adjust the number of channels in the output after convolution.

[0069] It should be noted that in the image processing model provided in this application embodiment, in each stage of the encoder except for the first stage (e.g. Figure 4 In the encoder, L2, L3, and L4 each correspond to a coarse skip connection. These skip connections are used to supplement the original features from the input image (the image to be processed). Specifically, they involve a series of operations on the input image (the image to be processed), such as mean pooling, pixel convolution, and depthwise convolution. Figure 4 The algorithm employs mean pooling, 1×1 pixel convolutions, and 3×3 depthwise convolutions. Through skip connections, low-level feature information can be directly passed to higher levels, allowing the encoder to more effectively focus on learning residual information (degradation information) caused by severe weather. This hybrid approach enables the image processing model to better utilize adaptive content in backgrounds unrelated to severe weather, as well as the characteristic features of weather degradation patterns. This facilitates the separation of unwanted degradation factors in potentially clear backgrounds, contributing to better image detail recovery.

[0070] Understandably, since each stage's output is obtained through upsampling, its C, H, and W values ​​will differ from the previous stage's output. Therefore, mean pooling can be used to downsample the image or feature map to be processed, 1×1 pixel convolution can be used to adjust the number of channels in the image or feature map, and 3×3 depthwise convolution can be used to extract the original features of the image or feature map. Furthermore, the feature map obtained through depthwise convolution can be concatenated (concat) with the feature map output from the previous stage (image processing module) along the channel dimension, that is, the two feature maps are stitched together, and the concatenated feature map is used as the input to the next stage (image processing module).

[0071] It should be noted that, since the image processing modules in the image processing model provided in this application embodiment contain the same components, have the same structure, and the same processing procedure, for ease of description, this application embodiment uses one image processing module as an example for explanation.

[0072] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an image processing module provided by an embodiment of the present application, as Figure 5 shown, each image processing module includes two key components: a dynamic-range histogram self-attention (DHSA) module and a dual-scale gated feed-forward (DGFF) module. The DHSA module and the DGFF module both interact with layer normalization (LN), the DHSA module can be used to capture the degradation factors caused by adverse weather in the dynamic distribution. The DGFF module can integrate multiple different range and different scale deep convolution paths in the transmission process, which can be used to extract multi-scale (two scales are taken as an example in the present application) features. LN can be used to normalize the output of each layer to ensure that each feature obeys consistent distribution between different layers, which can be used to solve the problem of gradient explosion or gradient disappearance, so that the model is more robust in the training process.

[0073] As Figure 5 shown, the image processing module also includes two skip connections, one skip connection can be used to supplement the features from the output of the previous stage, that is, the output of the previous stage image processing module can be pixel-wise summed with the output of the DHSA module (as shown by the symbol including a plus sign in the circle in Figure 5 ). Another skip connection can be used to supplement the features of the DHSA module output, that is, the output of the DHSA module can be pixel-wise summed with the output of the DGFF module to obtain the output of the image processing module. Specifically, the processing process of the image processing module can be as shown in formula 1 and formula 2:

[0074] F l = F l-1 + DHSA(LN(F l-1 )) formula 1

[0075] F l = F l + DGFF(LN(F l )) formula 2

[0076] In formula 1, F l represents the feature map of the stage (such as the lth stage) where the current image processing module is located, F l-1 represents the feature map output by the previous image processing module, DHSA represents the processing of the DHSA module, and LN represents the processing of layer normalization.

[0077] In formula 2, F on the left side of the equal sign represents the feature map output by the DHSA module in this stage (the image processing module), and F on the right side of the equal sign represents the F obtained by formula 1 l In formula 2, F on the left side of the equal sign represents the feature map output by the DHSA module in this stage (the image processing module), and F on the right side of the equal sign represents the F obtained by formula 1 l In formula 2, F on the left side of the equal sign represents the feature map output by the DHSA module in this stage (the image processing module), and F on the right side of the equal sign represents the F obtained by formula 1 l DGFF represents the processing of the DGFF module, and LN represents the processing of layer normalization.

[0078] Specifically, in order to better capture the degradation factors caused by adverse weather dynamically distributed in the image to be processed, the DHSA module can include a dynamic range convolution process and a self-attention mechanism. Next, the process of extracting image features of the image to be processed will be described in detail in combination with the dynamic range convolution process. Dynamic range convolution can be understood as a dynamic range convolution layer in an image processing model, which is a process of sorting feature elements in a feature map according to the size relationship of pixel values before convolution, and then performing convolution operation based on the sorted feature map, that is, reordering the input features between convolution operations. This is because the traditional convolution operation usually uses a fixed size convolution kernel, which limits the receptive field range, that is, the size of the region in the input image (the image to be processed or the feature map output by the previous layer) corresponding to each pixel in the feature map can be used to describe the perception range of the neuron in the network to the input image. If the receptive field is too small, only local and small range calculation can be performed, which may not be able to capture important features in the input image.

[0079] That is, the traditional convolution operation may be limited in this way. The sorting process can gather pixels with similar pixel values but uncertain positions (more dispersed) together, which is equivalent to reordering the spatial distribution of pixel value features, and then the convolution process can be understood as changing the fixed convolution range (such as 3x3 range) to a non-fixed (larger) convolution range, that is, dynamic range convolution.

[0080] In one possible implementation, the specific process of extracting image features of the image to be processed by the image processing device through the image processing model can include: performing first extraction processing on the image features of the image to be processed to obtain an initial feature map corresponding to the image to be processed, then performing sorting processing on the feature elements included in the initial feature map in the horizontal direction and the vertical direction according to the size relationship of the feature elements, to obtain an initial sorted feature map, and finally performing second extraction processing on the image features of the initial sorted feature map to obtain a feature map corresponding to the image to be processed.

[0081] The first extraction processing can refer to a convolution processing of the image processing model on the to-be-processed image or the feature map obtained in the intermediate process. The initial feature map is a feature map obtained through the first extraction processing. The feature elements included in the initial feature map can be understood as each pixel in the channel feature map corresponding to each channel of the initial feature map. The size relationship of each feature element is the size relationship of the pixel values of the pixels in the channel feature map. The initial ordering feature map is a feature map obtained through the ordering processing. The order of ordering can be ascending order of pixel values or descending order of pixel values, which is not limited in the present application. The second extraction processing can refer to a convolution processing of the image processing model on the image features of the initial ordering feature map. The size of the convolution kernel of the second extraction processing can be the same as or different from the size of the convolution kernel of the first extraction processing, which is not limited in the present application.

[0082] It should be noted that the first extraction processing described above can refer to a convolution processing performed by the image processing module in advance. In the case that the image processing module is an image processing module corresponding to the first stage in the encoder, as shown in L1 in the encoder in FIG. 6, the first extraction processing can be understood as a 3x3 convolution processing above the image processing module. The to-be-processed image is processed through 3x3 convolution to perform overlapping image block embedding. By using a stride smaller than the size of the convolution kernel, the adjacent convolution operations process partially repeated image blocks, thereby helping to preserve the local structure information of the to-be-processed image. In the case that the image processing module is any other image processing module, the first extraction processing can be understood as a 1x1 convolution processing before each image processing module, thereby obtaining the initial feature map. Figure 4

[0083] The most critical processing of the dynamic range convolution is to perform ordering processing on the feature elements included in the initial feature map in the horizontal direction and the vertical direction according to the size relationship of the feature elements included in the initial feature map, to obtain the initial ordering feature map. Since the initial feature map is a feature map including a plurality of channel feature maps corresponding to the respective channels, each pixel on each channel feature map can be sorted in ascending order of pixel values or descending order of pixel values in the row and column directions, respectively, to obtain the initial ordering feature map. In order to preserve part of the original local information, the feature elements in part of the channel feature maps can be sorted, which is also conducive to reducing the pressure on the video memory in the calculation process.

[0084] ​Specifically, the image processing device can perform channel equalization processing on the plurality of channel feature maps included in the initial feature map according to the image processing model, to obtain first type channel feature maps and second type channel feature maps; then, according to the size relationship of the feature elements included in the first type channel feature maps, the feature elements in each channel feature map included in the first type channel feature maps are sorted in the horizontal direction and the vertical direction to obtain first type channel sorted feature maps; finally, the first type channel sorted feature maps and the second type channel feature maps are concatenated in the corresponding channels to obtain an initial sorted feature map.

[0085] Wherein, taking the shape of the initial feature map as CxHxW as an example, C is the channel number of the initial feature map, and is an even number, for example, 36, 48, etc., so as to facilitate subsequent channel equalization processing. It should be noted that the above first extraction processing can also be used to adjust the channel number to an even number, for example, the channel number of the image to be processed is 3, and the first extraction processing can be used to adjust the channel number of the initial feature map to an even number, such as 36, 48, etc. The channel equalization processing on the plurality of channel feature maps included in the initial feature map can be understood as splitting the initial feature map of CxHxW along the channel dimension into two feature maps of C / 2xHxW, i.e. the first type channel feature maps and the second type channel feature maps, the first type channel feature maps can include channel feature maps with serial numbers from 1 to C / 2, and the first type channel feature maps can include channel feature maps with serial numbers from C / 2+1 to C. For example, taking C of the initial feature map as 36 as an example, after channel equalization processing, the first type channel feature maps including channel feature maps with serial numbers from 1 to 18 and the second type channel feature maps including channel feature maps with serial numbers from 19 to 36 can be obtained.

[0086] Then, the feature elements in each channel feature map included in the first type channel feature maps are sorted in the horizontal direction and the vertical direction. If each channel feature map (such as each of the 18 channel feature maps) is regarded as a matrix, the feature elements in each row and each column are sorted to obtain the first type channel sorted feature maps, and each channel feature map in the first type channel feature maps is sorted, for example, including 18 sorted channel feature maps. Thus, the sorting processing can organize the pixels with lower and higher pixel values into the diagonal corners of the matrix in a regular pattern, for example, sorted in ascending order of pixel values, and the top left corner of the matrix of each channel feature map in the first type channel sorted feature maps is the pixel with the smallest pixel value, and the bottom right corner of the matrix is the pixel with the largest pixel value. The pixels with smaller pixel values and the pixels with larger pixel values can be stacked on the diagonal of each channel feature map.

[0087] Further, the first type of channel ordering feature map and the second type of channel feature map are processed in series along the channel dimension, that is, the ordered first type of channel ordering feature map and the unsorted second type of channel feature map are concatenated along the channel dimension to obtain an initial ordering feature map. For example, the above-mentioned channel feature maps with serial numbers 1-18 are spliced with the channel feature maps with serial numbers 19-36 in order of serial number to obtain an initial ordering feature map with a channel number of 36.

[0088] Further, the image features of the initial ordering feature map can be subjected to a second extraction processing, which can be a convolution processing based on the sorted channel feature map. Thus, through the sorting process, the convolution can be allowed to perform calculation in a dynamic range to obtain a feature map. Since different adverse weather causes degradation factors to exhibit similar patterns in the degraded image, the degraded pixels are often concentrated in adjacent spatial positions and separated from clear pixels. After sorting, the degraded pixels can be gathered together, and the clear pixels can also be gathered together. Convolution processing under such arrangement enables the convolution kernel to focus on extracting degraded features and clear features, so as to subsequently focus on preserving clear information and restoring degraded features.

[0089] For example, the initial feature map F obtained through the first extraction processing For explanation, it is represented that the initial feature map F is a three-dimensional tensor, the elements of which belong to a real number set R, and the shape of the tensor is CxHxW. First, the C channel feature maps are subjected to channel equalization processing, that is, split into two branches along the channel to obtain a first type of channel feature map F1 and a second type of channel feature map F2. Further, the feature elements in each channel feature map included in the first branch F1 are sorted in the horizontal direction and the vertical direction to obtain a first type of channel ordering feature map. Finally, the first type of channel ordering feature map and the second type of channel feature map are concatenated along the channel dimension to obtain a feature map F. Specifically, the processing process of the dynamic range convolution can be as shown in formula 3, formula 4 and formula 5:

[0090] F1,F2=Split(F) Formula 3

[0091] F1=Sort v (Sort h (F1)) Formula 4

[0092]

[0093] In formula 3, F represents the initial feature map Split() is a split function, indicating that the channel dimension is evenly divided, F1 and F2 represent two branches obtained by splitting, F1 represents the first type of channel feature map, and F2 represents the second type of channel feature map.

[0094] In formula 4, F1 on the left side of the equal sign represents the first type of channel sorting feature map, Sort v () and Sort h () are sorting functions, Sort v () represents sorting along the vertical direction, that is, sorting according to the size relationship of each column of feature elements, Sort h () represents sorting along the horizontal direction, that is, sorting according to the size relationship of each row of feature elements, and F1 on the right side of the equal sign represents the first type of channel feature map.

[0095] In formula 5, F on the left side of the equal sign represents the feature map, that is, the processing result obtained by dynamic range convolution, represents a 3x3 depthwise convolution (Depthwise Convolution), Conv 1×1 represents a 1x1 pointwise convolution (Pointwise Convolution), and these two parts are components of a separable convolution (Separable Convolution). That is, the recombined feature map is subjected to subsequent separable convolution processing. Separable convolution refers to splitting the standard convolution operation into multiple small convolution kernels and performing separation operation in the spatial dimension, thereby reducing the number of multiplication calculations and reducing the computational complexity. Depthwise convolution uses only one convolution kernel to convolve the input channels one by one, that is, each channel feature map is independently subjected to spatial convolution. Pointwise convolution can use a 1x1 convolution kernel to mix between channels, such as changing the number of channels of the feature map or performing linear transformation processing on the feature map. In this way, depthwise separable convolution can significantly reduce the number of parameters and reduce the computational complexity while maintaining certain feature extraction capability. F1 on the right side of the equal sign represents the first type of channel sorting feature map, F2 represents the second type of channel feature map, and Concat() is a concatenation function that can connect F1 and F2 along the channel dimension.

[0096] wherein the initial sorting feature map is the result obtained by Concat(F1, F2), and the second extraction processing can refer to the above separable convolution, such as including a 3x3 depthwise convolution and a 1x1 pointwise convolution. Thus, the initial sorting feature map is subjected to the second extraction processing to obtain a feature map.

[0097] S202, according to the size relationship of each feature element included in the above feature map, sorting the feature elements included in the above feature map to obtain a sorting feature map.

[0098] In the embodiment of the present application, the feature map is obtained by performing second extraction processing on the initial ordering feature map, and the initial ordering feature map is obtained by performing channel concatenation processing based on the ordered first type channel ordering feature map and the unordered second type channel feature map. The feature map also includes channel feature maps corresponding to multiple channels, and each feature element in the feature map can refer to a feature element in each channel feature map included in the feature map. The ordering of the feature elements in the feature map according to the size relationship of the feature elements refers to ordering processing in ascending order of pixel values or descending order of pixel values, and the feature map including the ordering result is an ordering feature map.

[0099] It can be understood that, since the DHSA module is used to implement the self-attention mechanism of the image processing model, in order to determine the feature representations of Query (Q), Value (V) and Key (K), the feature map can be split into multiple sub-feature maps, such as 3, along the channel dimension. Then, Q, K and V can be determined based on the multiple sub-feature maps respectively to determine the attention map. It should be noted that the 1x1 point-by-point convolution in the second extraction processing (separable convolution) can be used to adjust the number of channels of the feature map. In order to subsequently divide the feature map along the channel dimension into multiple sub-feature maps, the number of channels of the feature map can be adjusted to an integer multiple of the number of sub-feature maps to be divided, such as an integer multiple of 3, for example, 36, 48, etc., so as to facilitate subsequent channel division processing.

[0100] In a possible implementation, the specific process of the ordering processing performed by the image processing device on the feature map through the image processing model can be: according to the multiple channel feature maps included in the feature map, performing channel division processing on the multiple channel feature maps to obtain multiple types of channel feature maps; and then, according to the size relationship of the feature elements included in the multiple types of channel feature maps, respectively ordering the feature elements included in the multiple types of channel feature maps to obtain ordering feature maps corresponding to the multiple types of channel feature maps respectively. The multiple types of channel feature maps represent multiple sub-feature maps containing multiple channel feature maps obtained by channel division processing, for example, the shape of the feature map is 3N×H×W, and if the channel division processing is to split the feature map into 3 sub-feature maps, the multiple types of channel feature maps refer to the 3 sub-feature maps, and the shape of each sub-feature map is N×H×W. Each feature element included in the multiple types of channel feature maps can be a feature element in each channel feature map included in each sub-feature map, such as a feature element in N channel feature maps included in the sub-feature map with the size of N×H×W.

[0101] The three sub-feature maps included in the multi-class channel feature map can specifically be a third-class channel feature map, a fourth-class channel feature map, and a fifth-class channel feature map. In order to subsequently assign different attentions based on multiple dimensions, the application takes two dimensions as an example for explanation. The three sub-feature maps can be taken as the feature representation of V and the feature representation of two pairs of QK, for example, the third-class channel feature map is taken as the feature representation of V, the fourth-class channel feature map is taken as the joint feature representation of Q1K1, and the fifth-class channel feature map is taken as the joint feature representation of Q2K2. Then, the three sub-feature maps can be respectively transmitted to two branches to generate attention maps of two dimensions.

[0102] According to the size relationship of each feature element included in the multi-class channel feature map, each feature element included in the multi-class channel feature map is sorted and processed, which means that each sub-feature map (such as the third-class channel feature map, the fourth-class channel feature map, and the fifth-class channel feature map) is sorted and processed according to the ascending order of the pixel value or the descending order of the pixel value of each channel feature map included in the sub-feature map, to obtain the sorted feature map corresponding to the multi-class channel feature map, that is, to obtain the sorted feature map corresponding to each sub-feature map. It should be noted that although the feature map is obtained by sorting and processing, the order of the feature elements in each channel feature map can be disturbed after the sorting and processing (i.e., after obtaining the initial sorted feature map) and the second extraction processing, and therefore, the sorted processing can be performed on the sub-feature maps obtained by splitting after the channel division processing. The sorting order can be consistent with the sorting order of the sorting and processing performed to obtain the initial sorted feature map, so as to obtain the sorted feature map corresponding to the multi-class channel feature map.

[0103] Specifically, in order to reduce the computing pressure of the display memory, the image processing device can perform feature reorganization processing on the feature elements in the multiple channel feature maps included in the multi-class channel feature map before sorting and processing the feature elements included in the multi-class channel feature map, so as to convert the multi-class channel feature map from a three-dimensional tensor to a two-dimensional tensor, to obtain the channel conversion feature map corresponding to the multi-class channel feature map, and then sort and process the channel conversion feature map corresponding to the multi-class channel feature map according to the size relationship of each feature element included in the channel conversion feature map, to obtain the channel sorted feature map corresponding to the multi-class channel feature map.

[0104] The multiple channel feature maps included in each of the multiple types of channel feature maps can be the channel feature maps included in each of the sub-feature maps obtained by splitting the feature map, and specifically can refer to the multiple channel feature maps included in the third type of channel feature map, the fourth type of channel feature map, and the fifth type of channel feature map. The reorganization processing can also be referred to as reshape, which means that the shape of the feature elements in each sub-feature map is changed, i.e., converted from a three-dimensional tensor to a two-dimensional tensor, by recombining the feature elements. The sub-feature maps after shape change are the channel conversion feature maps corresponding to the multiple types of channel feature maps, i.e., the third type of channel feature map, the fourth type of channel feature map, and the fifth type of channel feature map after shape change. Then, based on the size relationship of the feature elements included in the third type of channel feature map, the fourth type of channel feature map, and the fifth type of channel feature map after shape change, sorting processing is performed, and the sorting result is the channel ordering feature map corresponding to the multiple types of channel feature maps.

[0105] The reorganization operation (reshape operation) is generally used to change the shape of the tensor for subsequent calculation, such as a specific matrix multiplication operation. The reshape operation does not change the data (feature elements) in the tensor, but only changes the arrangement of the data. For example, the shape of the third type of channel feature map, the fourth type of channel feature map, and the fifth type of channel feature map is CxHxW. Through reorganization processing, the spatial dimensions (H and W) are combined into one dimension, and the result of the reorganization processing is obtained, i.e., the channel conversion feature map corresponding to the multiple types of channel feature maps is obtained, and the shape of each channel conversion feature map is CxHW. For example, the shape of the third type of channel feature map, the fourth type of channel feature map, and the fifth type of channel feature map is 3x4x5. After reorganization processing of the third type of channel feature map, the fourth type of channel feature map, and the fifth type of channel feature map, the shape of the channel conversion feature map is 3x20. In this process, the feature elements in each channel feature map are continuously arranged into a one-dimensional array according to their order in the channel feature map, and the channel dimension remains unchanged.

[0106] Further, the channel conversion feature maps respectively corresponding to the multi-class channel feature maps can be respectively sorted. Specifically, the sorting process can first sort the feature elements in each channel feature map included in the specified class channel conversion feature map according to the size relationship of the feature elements included in the specified class channel conversion feature map in the channel conversion feature maps respectively corresponding to the multi-class channel feature maps, to obtain a specified class channel sorting feature map and a sorting index. Further, the channel conversion feature maps other than the specified class channel conversion feature map in the channel conversion feature maps respectively corresponding to the multi-class channel feature maps are sorted according to the sorting index. The specified class channel conversion feature map can be understood as a sub-feature map in the multi-class channel feature map, the sorting index is used to indicate the element position of the feature elements included in the specified class channel sorting feature map in the specified class channel conversion feature map, and the channel conversion feature maps other than the specified class channel conversion feature map in the multi-class channel feature map are sub-feature maps other than the sorted sub-feature map in the multi-class channel feature map, which are unsorted sub-feature maps.

[0107] It can be understood that this process means that the image processing device can first sort the feature elements in one of the sub-feature maps (the specified class channel conversion feature map) in ascending order of pixel value or descending order of pixel value, to obtain a sorting result (the specified class channel sorting conversion feature map) and a sorting index. Then, the image processing device only needs to sort the other channel conversion feature maps in the multi-class channel feature map according to the sorting index. That is, the feature elements in the other channel conversion feature maps are arranged according to the positions indicated by the sorting index, to obtain the channel sorting feature maps respectively corresponding to the multi-class channel feature maps. The process of sorting the other channel conversion feature maps based on the sorting index is implemented by the Gather() function, so the process of sorting the other channel conversion feature maps based on the sorting index can also be called aggregation processing.

[0108] For example, the channel conversion feature map corresponding to the multi-class channel feature map includes A = [3, 1, 2] and B = [b1, b2, b3], where A and B each represent a one-dimensional array (vector) in a channel of the respective channel conversion feature map. First, A (the specified class channel conversion feature map) can be sorted in ascending order of pixel values to obtain a specified class channel sorted feature map, i.e., A_sorted = [1, 2, 3]. The sorting index that enables the sorting is [1, 2, 0], because the smallest number 1 is in the second position in A, and the index of the smallest value starts from 0 in computer language, so the corresponding index is 1, and the largest number 3 is in the first position in A, and the index of the smallest value starts from 0, so the corresponding index is 0. Thus, B (the other channel conversion feature map) can be sorted in the same way as A based on the sorting index to obtain B_sorted = [b2, b3, b1], and A_sorted and B_sorted are referred to as the channel sorted feature maps corresponding to the multi-class channel feature map, respectively.

[0109] In the above sorting process, the third class channel conversion feature map corresponding to the third class channel feature map (the feature representation of V) can be sorted first, i.e., the specified class channel conversion feature map is the third class channel conversion feature map, to obtain a third class channel sorted feature map and a sorting index, and then the fourth class channel conversion feature map corresponding to the fourth class channel feature map (the feature representation of Q1K1) and the fifth class channel conversion feature map corresponding to the fifth class channel feature map (the feature representation of Q2K2) can be sorted based on the sorting index, respectively, to obtain a fourth class channel sorted feature map and a fifth class channel sorted feature map. The third class channel sorted feature map, the fourth class channel sorted feature map, and the fifth class channel sorted feature map constitute a sorted feature map, i.e., the channel sorted feature maps corresponding to the above multi-class channel feature map, respectively.

[0110] S203, performing first feature reorganization processing on the sorted feature map according to the preset number of feature element categories to obtain a first reorganized feature map, and performing second feature reorganization processing on the sorted feature map according to the preset number of elements under each feature element category to obtain a second reorganized feature map.

[0111] In the embodiments of the present application, the first feature reorganization processing and the second feature reorganization processing can also be referred to as reshape processing, which means that the feature elements in the sorted feature map are reorganized by different recombination methods, and are converted from a two-dimensional tensor to two three-dimensional tensors with different shapes, and the reshape results are the first reorganized feature map and the second reorganized feature map. The preset number of feature element categories and the preset number of elements under each feature element category are, respectively, the height parameter H in the three-dimensional tensor CxHxW converted by the first feature reorganization processing and the second feature reorganization processing, and the value of the height parameter H is a preset numerical parameter. It should be noted that the H in the three-dimensional tensor obtained after the two feature reorganization processes represents the height parameter, but the values are different, and the meanings are also different. In order to perform parallel computing and achieve load balancing, the number of elements under each feature element category in the embodiments of the present application can be the same.

[0112] The sorted feature map includes a third type of channel sorted feature map, a fourth type of channel sorted feature map and a fifth type of channel sorted feature map, the third type of channel sorted feature map corresponds to the feature representation of V, the fourth type of channel sorted feature map corresponds to the joint feature representation of Q1K1, and the fifth type of channel sorted feature map corresponds to the joint feature representation of Q2K2. Before the two different reorganization processes are specifically performed, the fourth type of channel sorted feature map can be first subjected to a splitting process along the channel dimension and the fifth type of channel sorted feature map can be first subjected to a splitting process along the channel dimension, to obtain Q1, K1, Q2 and K2. Specifically, according to the plurality of channel feature maps included in the fourth type of channel sorted feature map, the plurality of channel feature maps included in the fourth type of channel sorted feature map are subjected to channel equalization processing to obtain a first sub-type channel sorted feature map and a second sub-type channel sorted feature map, and according to the plurality of channel feature maps included in the fifth type of channel sorted feature map, the plurality of channel feature maps included in the fifth type of channel sorted feature map are subjected to channel equalization processing to obtain a third sub-type channel sorted feature map and a fourth sub-type channel sorted feature map.

[0113] The first sub-type channel sorted feature map can correspond to the feature representation of Q1, the second sub-type channel sorted feature map can correspond to the feature representation of K1, the third sub-type channel sorted feature map can correspond to the feature representation of Q2, and the fourth sub-type channel sorted feature map can correspond to the feature representation of K2. Taking the shapes of the fourth type of channel sorted feature map and the fifth type of channel sorted feature map as CxHW for example, the channel equalization processing can be equalization along the channel dimension to obtain feature maps with a shape of C / 2xHW, i.e., the first sub-type channel sorted feature map, the second sub-type channel sorted feature map, the third sub-type channel sorted feature map and the fourth sub-type channel sorted feature map.

[0114] Specifically, the specific process of determining V, Q1, K1, Q2 and K2 according to the feature map can be as shown in formulas 6-8:

[0115]

[0116]

[0117]

[0118] In formula 6, denotes the reshaping of the feature from to i.e. the shape is converted from CxHxW to CxHW. The Sort() function denotes sorting, V denotes the feature representation of the value, and d denotes the index of the feature element after the sorting processing, i.e. the sorting index mentioned above.

[0119] In formula 7 and formula 8, also denotes the shape conversion from CxHxW to CxHW, F QK,1 denotes the joint feature representation of Q1K1, i.e. the fourth type of channel feature map mentioned above. F QK,2 denotes the joint feature representation of Q2K2, i.e. the fifth type of channel feature map mentioned above. Gather() is an aggregation function, which denotes the operation of retrieving elements from a tensor according to a given sorting index d, i.e. the operation of sorting processing on other channel conversion feature maps (the fourth type of channel conversion feature map and the fifth type of channel conversion feature map) based on the sorting index. Split() is a splitting function, which can denote channel equalization processing, i.e. splitting processing along the channel dimension, splitting the joint feature representation of Q1K1 into the feature representations of Q1 and K1, and splitting the joint feature representation of Q2K2 into the feature representations of Q2 and K2.

[0120] Further, the two branches can be reorganized respectively, specifically, the third type of channel sorting feature map V, the first sub-type of channel sorting feature map Q1 and the second sub-type of channel sorting feature map K1 can be subjected to first feature reorganization processing according to a preset number of feature element categories, to obtain a first reorganized feature map, and the third type of channel sorting feature map V, the third sub-type of channel sorting feature map Q2 and the fourth sub-type of channel sorting feature map K2 can be subjected to second feature reorganization processing according to a preset number of elements, to obtain a second reorganized feature map.

[0121] Specifically, the specific process of the first feature reorganization processing can be that, according to the feature element category number, feature elements in each channel feature map included in the third class channel ordering feature map, the first sub-class channel ordering feature map and the second sub-class channel ordering feature map are subjected to feature reorganization processing to convert the third class channel ordering feature map, the first sub-class channel ordering feature map and the second sub-class channel ordering feature map from two-dimensional tensors into three-dimensional tensors respectively, to obtain a third class channel conversion feature map, a first sub-class channel conversion feature map and a second sub-class channel conversion feature map; and then, according to the third class channel conversion feature map, the first sub-class channel conversion feature map and the second sub-class channel conversion feature map, the first reorganization feature map is generated.

[0122] wherein the feature element category number is a preset parameter, the height parameter of the converted three-dimensional tensor, and the width parameter of the three-dimensional tensor is the number of elements in each feature element category determined according to the number of elements in each channel feature map included in the third class channel conversion feature map, the first sub-class channel conversion feature map and the second sub-class channel conversion feature map and the feature element category number, that is, the number of elements in each feature element category can be determined according to the number of feature elements included in the channel feature map under each channel and the preset feature element category number. The first reorganization feature map includes the third class channel conversion feature map, the first sub-class channel conversion feature map and the second sub-class channel conversion feature map, and the first sub-class channel conversion feature map and the second sub-class channel conversion feature map are obtained based on the fourth class channel feature map through a series of processing. The feature element category can be referred to as a distribution bin for classifying pixels, and the first reorganization processing can also be referred to as bin-wise histogram reshaping (BHR).

[0123] Exemplarily, taking the shapes of the third-type channel conversion feature map V, the first-type sub-channel conversion feature map Q1, and the second-type sub-channel conversion feature map K1 as CxHW, the two-dimensional tensor with the shape of CxHW is converted into a three-dimensional tensor with the shape of CxBxHW / B through the first reorganization processing, where B is a preset feature element category number, which can also be referred to as a number of distribution map boxes, and HW / B is the number of elements in each feature element category determined according to the number of elements (i.e., the number of feature elements) in each channel feature map in V, Q1, and K1 and B, which is a width parameter of the three-dimensional tensor. It can be understood that B can be set to a smaller value in BHR, and in the case that the number of feature elements in V, Q1, and K1 is certain, HW / B is larger, that is, a large number of pixels including dynamic positions in each feature element category (distribution map box), and then the global information (comprehensive features) in a larger range (global scale) can be extracted based on V, Q1, and K1 after the sorting processing, that is, pixels covering a wider range of pixel values, so that a large range of information can be extracted, and the integration of global features can be promoted.

[0124] Please refer to Figure 6 , Figure 6 is a principle diagram of feature element classification provided by an embodiment of the present application, as shown in Figure 6 , the three images on the top can represent image blocks in a rain and fog scene, a raindrop scene, and a snowy scene, respectively. The degradation factors caused by bad weather will lead to similar patterns between pixels, that is, the pixel values of the degraded pixels are similar, and after the sorting processing, the first reorganization processing is performed again to obtain the first reorganization feature map, which can classify the pixels of the image pattern affected by the weather into one or more feature element categories (distribution map boxes) to a certain extent, and classify the pixels (clear pixels) of the background image not affected by the weather into another one or more feature element categories. The preset number of feature element categories is B, as shown in Figure 6 , B categories (distribution map boxes), each of which is either a clear pixel or a degraded pixel.

[0125] Specifically, the specific process of the second feature reorganization processing can be that the feature elements in each channel feature map included in the third-type channel sorting feature map, the third-type sub-channel sorting feature map, and the fourth-type sub-channel sorting feature map are subjected to feature reorganization processing according to the preset number of elements in each feature element category, so as to convert the third-type channel sorting feature map, the third-type sub-channel sorting feature map, and the fourth-type sub-channel sorting feature map from two-dimensional tensors into three-dimensional tensors, respectively, to obtain the third-type channel conversion feature map, the third-type sub-channel conversion feature map, and the fourth-type sub-channel conversion feature map, and then generate the second reorganization feature map according to the third-type channel conversion feature map, the third-type sub-channel conversion feature map, and the fourth-type sub-channel conversion feature map.

[0126] wherein the element quantity in each feature element category is a preset parameter, the height parameter of the converted three-dimensional tensor is the preset element quantity in each feature element category, and the width parameter of the three-dimensional tensor is the number of feature element categories determined according to the element quantity in each channel feature map of the third type channel converted feature map, the third sub-type channel converted feature map and the fourth sub-type channel converted feature map, and the preset element quantity in each feature element category. That is, the number of feature element categories can be determined according to the number of feature elements included in the channel feature map under each channel, and the preset element quantity in each feature element category. The second reorganization feature map includes the third type channel converted feature map, the third sub-type channel converted feature map and the fourth sub-type channel converted feature map, and the third sub-type channel converted feature map and the fourth sub-type channel converted feature map are obtained based on a series of processing of the fifth type channel feature map. The second reorganization processing can also be referred to as frequency-wise histogram reshaping (FHR).

[0127] For example, the shapes of the third type channel converted feature map V, the third sub-type channel converted feature map Q2 and the fourth sub-type channel converted feature map K2 are CxHW, and the two-dimensional tensor with the shape of CxHW is converted into a three-dimensional tensor with the shape of CxBxHW / B through the second reorganization processing, wherein B is the preset element quantity in each feature element category, and HW / B is the number of feature element categories determined according to the element quantity (i.e. the number of feature elements) in each channel feature map of V, Q2 and K2, and the frequency B of each histogram bin, that is, the number of distribution bins, which is the width parameter of the three-dimensional tensor. It can be understood that in FHR, B can be set to a small value, and in the case that the number of feature elements of V, Q2 and K2 is certain, HW / B is large, that is, the number of feature element categories is large, and the number of feature elements in each feature element category is small, so that local information can be extracted based on V, Q2 and K3 after sorting processing, so that each feature element category is more focused on a small number of pixels with similar pixel values, thereby extracting fine information and enhancing the practicability of more detailed features.

[0128] It should be noted that the first reorganization processing and the second reorganization processing both convert the two-dimensional tensor with the shape of CxHW into a three-dimensional tensor with the shape of CxBxHW / B, but the meanings of B and HW / B in the two are different. It can be understood that the first reorganization processing is to extract global information in a large range, and the second reorganization processing is to extract fine information, so the element quantity in each feature element category corresponding to the first reorganization feature map is greater than the element quantity in each feature element category corresponding to the second reorganization feature map.

[0129] Please refer to Figure 7 and Figure 8 , Figure 7A principle diagram of a first reorganization processing provided in an embodiment of the present application, Figure 8 A principle diagram of a second reorganization processing provided in an embodiment of the present application, Figure 7 and Figure 8 As shown in the left side of the figure, before the first reorganization processing and the second reorganization processing, the tensor example is in the shape of CxHW. After the first reorganization processing, each column on the right side can be regarded as a feature element category (histogram bin), and the columns constitute B feature element categories. The feature elements in each column are the feature elements under the feature element category, and the number of feature elements under each feature element category is HW / B. Similarly, after the second reorganization processing, each row on the right side can be regarded as a feature element category (histogram bin), and the rows constitute HW / B feature element categories. The number of feature elements in each row is B, which is the feature elements under the feature element category. Figure 7 Figure 8

[0130] Thus, the attention map can be generated according to the first reorganization feature map including the third type of channel conversion feature map, the first sub-type of channel conversion feature map and the second sub-type of channel conversion feature map, and the second reorganization feature map including the third type of channel conversion feature map, the third sub-type of channel conversion feature map and the fourth sub-type of channel conversion feature map.

[0131] S204, generate an attention map according to the first reorganization feature map and the second reorganization feature map, and perform a reconstruction processing on the to-be-processed image according to the attention map, to obtain a recovery image corresponding to the to-be-processed image.

[0132] In the embodiment of the present application, the attention map refers to a mechanism for describing the importance of related pixels in an image (such as a to-be-processed image), which can be understood as a mechanism including the attention weight of each pixel, a mechanism for dynamically balancing the importance of different parts of the input image (to-be-processed image) of the image processing model, and a mechanism for determining which parts should be paid attention to. The reconstruction processing refers to the recovery processing of the image based on the attention map, which can be used to reconstruct the degraded image affected by the bad weather, so as to obtain a clean image without the degradation factors, i.e., the recovery image corresponding to the to-be-processed image.

[0133] ​​In a possible implementation, the process of generating the attention map can specifically be that a first attention map is generated according to the third type of channel conversion feature map, the first sub-type of channel conversion feature map and the second sub-type of channel conversion feature map, and a second attention map is generated according to the third type of channel conversion feature map, the third sub-type of channel conversion feature map and the fourth sub-type of channel conversion feature map, and then the first attention map and the second attention map are fused to obtain the attention map. The third type of channel conversion feature map corresponds to the feature representation of V, the first sub-type of channel conversion feature map and the second sub-type of channel conversion feature map correspond to the feature representations of Q1 and K1 respectively, and the third sub-type of channel conversion feature map and the fourth sub-type of channel conversion feature map correspond to the feature representations of Q2 and K2 respectively.

[0134] Specifically, the specific process of generating the first attention map can include first fusion processing of the first sub-type of channel conversion feature map and the second sub-type of channel conversion feature map to obtain a first fusion feature map, and then first normalization processing of the first fusion feature map and the third type of channel conversion feature map to obtain the first attention map. The specific process of generating the second attention map can include second fusion processing of the third sub-type of channel conversion feature map and the fourth sub-type of channel conversion feature map to obtain a second fusion feature map, and then second normalization processing of the second fusion feature map and the third type of channel conversion feature map to obtain the second attention map.

[0135] Specifically, the first fusion processing can be fusion processing of Q1 and K1, the second fusion processing can be fusion processing of Q2 and K2, the first normalization processing can be fusion processing of the first fusion feature map and V based on a normalization function, and the second normalization processing can be fusion processing of the second fusion feature map and V based on a normalization function. Specifically, the above Q1, K1 and Q2, K2 as two branches, respectively pass through two types of reorganization processing, and respectively pass through self-attention processes, and the outputs of the two self-attention processes are multiplied element by element to generate the process of the final output attention map, which can be referred to as formulas 9-11:

[0136]

[0137]

[0138] A=A B ⊙A F Formula 11

[0139] In formulas 9-11, A B represents an attention map generated based on BHR, that is, the first attention map, A F an attention map generated based on FHR, that is, the second attention map, R Bdenotes the recombination processing corresponding to the BHR, i.e., the first recombination processing, R F denotes the recombination processing corresponding to the FHR, i.e., the second recombination processing. That is, after V, Q1, K1, Q2, K2 are determined, in the BHR, a self-attention mechanism can be performed on the basis of the dimension of the feature element category (distribution box), i.e., the dimension of B, to perform a larger range of feature extraction, and in the FHR, a self-attention mechanism can be performed on the basis of the dimension of the number of elements (frequency) under each feature element category, i.e., also the dimension of B, to perform fine feature extraction. T denotes the transpose of a matrix, and k is the number of heads in the multi-head self-attention mechanism. In each image processing module, the value of k can be different or the same, which is not limited in the present application, and can be determined on the basis of the model structure designed according to the actual application scenario. In the embodiments of the present application, the value of k can be 2, 4, 8, etc.

[0140] wherein Q1, K1, Q2, K2 and V are the first sub-class channel conversion feature map, the second sub-class channel conversion feature map, the third sub-class channel conversion feature map, the fourth sub-class channel conversion feature map and the third-class channel conversion feature map respectively. Softmax is a normalization function or formula, which can be used to compress (map) a k-dimensional real number vector into another k-dimensional real number vector. Each element of the mapped real number vector can have a value between 0 and 1, and the sum of all elements is 1. ⊙ denotes pixel-wise multiplication, and A denotes an attention map, i.e., the final attention map output, thereby completing the combination of global and local dynamic feature aggregation and the double-channel distribution map self-attention mechanism.

[0141] It can be understood that, since the feature map is obtained on the basis of sorting processing, after the attention map is obtained, the sorted attention map needs to be restored. Specifically, the shape of the attention map is C×G×W, and then the multiple channel feature maps included in the attention map can be subjected to channel equalization processing to obtain a first-class channel attention map and a second-class channel attention map. Then, the first-class channel attention map and the second-class channel attention map are subjected to restoration processing according to the reference sorting index to obtain a first restored attention map and a second restored attention map, and finally the first restored attention map and the second restored attention map are subjected to corresponding channel concatenation processing to obtain a target attention map. The shape of the first-class channel attention map and the second-class channel attention map can be C / 2×H×W, and the principle of channel equalization processing can be referred to the related description in step S201, which will not be described herein. The reference sorting index is used to indicate the element position of each feature element included in the first-class channel sorting feature map in the first-class channel feature map, i.e., the sorting index obtained by initially performing sorting processing on the initial feature map in the horizontal direction and the vertical direction.

[0142] Understandably, this restoration process can involve reordering the feature elements in the feature maps of the first and second channel attention maps according to their positions in the reference sorting index, resulting in a first restored attention map and a second restored attention map. Then, the first and second restored attention maps are concatenated along their corresponding channels to obtain a target attention map of shape C×H×W. The principle of this concatenation process can be found in the relevant description in step S201, and will not be repeated here.

[0143] The above is the entire processing procedure of the DHSA module. The following section will combine... Figure 9 This section provides an overview of the DHSA module's processing procedure. Please refer to the following: Figure 9 , Figure 9 This is a processing timing diagram of a DHSA module provided in an embodiment of this application, such as... Figure 9 As shown, the first step is to perform dynamic range convolution. The input to this dynamic range convolution is an initial feature map of shape C×H×W. This map is then divided into two feature maps of shape C / 2×H×W: the first type of channel feature map and the second type of channel feature map. Next, the first type of channel feature map is sorted horizontally and vertically to obtain the sorted first type of channel feature map and a reference sorting index, as shown below. Figure 9 The matrix containing numbers is shown to the right of the sorting box. Then, the first and second channel sorting feature maps are concatenated along the channel dimension to obtain a feature map of shape C×H×W, i.e., the initial sorting feature map. Finally, this initial sorting feature map undergoes a second extraction process, namely, separable convolution (1×1 pointwise convolution and 3×3 depthwise convolution), to obtain another feature map of shape C×H×W.

[0144] After dynamic range convolution processing, the distribution map self-attention process can be performed. The input to this distribution map self-attention is a feature map of shape C×G×W. The input feature map is then subjected to channel-wise equalization, resulting in three feature maps of shape C / 3×H×W: the third-class channel feature map, the fourth-class channel feature map, and the fifth-class channel feature map. For ease of description, Figure 9Still taking the shape of the feature map obtained by processing the channel as CxHxW for explanation, the actual number of channels is one third of the former. The third type of channel feature map is taken as the feature representation of V, and the fourth type of channel feature map and the fifth type of channel feature map are taken as the joint feature representation of Q1K1 and the joint feature representation of Q2K2 respectively. Reshaping is performed on the third type of channel feature map, the fourth type of channel feature map and the fifth type of channel feature map respectively, that is, the shape of the three is converted from a three-dimensional tensor to a two-dimensional tensor, and three feature maps with a shape of CxHW are obtained, that is, the fourth type of channel conversion feature map Q1K1, the fifth type of channel conversion feature map Q2K2 and the third type of channel conversion feature map V.

[0145] Further, V (specified type channel conversion feature map) with a shape of CxHW is sorted first to obtain a sorting result (i.e. the third type of channel sorting feature map) and a sorting index, and the sorting index is transmitted to Q1K1, Q2K2, i.e. to other channel conversion feature maps. Based on the sorting index, the other channel conversion feature maps are sorted (aggregated) to obtain feature maps with a shape of CxHW, i.e. the fourth type of channel sorting feature map and the fifth type of channel sorting feature map. Then, the fourth type of channel sorting feature map Q1K1 and the fifth type of channel sorting feature map Q2K2 are respectively processed by channel equalization to obtain feature maps with a shape of C / 2xHW, i.e. the first sub-type channel conversion feature map Q1, the second sub-type channel conversion feature map K1, the third sub-type channel conversion feature map Q2 and the fourth sub-type channel conversion feature map K2. It should be noted that, in order to facilitate description, Figure 9 Still taking the shape of the feature map obtained by processing the channel as CxHW for explanation, the actual number of channels is one third of the former. The third type of channel feature map is taken as the feature representation of V, and the fourth type of channel feature map and the fifth type of channel feature map are taken as the joint feature representation of Q1K1 and the joint feature representation of Q2K2 respectively. Reshaping is performed on the third type of channel feature map, the fourth type of channel feature map and the fifth type of channel feature map respectively, that is, the shape of the three is converted from a three-dimensional tensor to a two-dimensional tensor, and three feature maps with a shape of CxHW are obtained, that is, the fourth type of channel conversion feature map Q1K1, the fifth type of channel conversion feature map Q2K2 and the third type of channel conversion feature map V.

[0146] Further, respectively, first reshaping (BHR) is performed on Q1, K1 and V, and second reshaping (FHR) is performed on Q2, K2 and V to obtain feature maps with a shape of CxBxHW / B, i.e. the first reshaped feature map including the third type of channel conversion feature map V, the first sub-type channel conversion feature map Q1 and the second sub-type channel conversion feature map K2, and the second reshaped feature map including the third type of channel conversion feature map V, the third sub-type channel conversion feature map Q2 and the fourth sub-type channel conversion feature map K2. Matrix multiplication (i.e. first fusion processing) is performed on Q1 and K1 to obtain a first fusion feature map, and similarly, matrix multiplication (i.e. second fusion processing) is performed on Q2 and K2 to obtain a second fusion feature map, as shown in the matrix image of the middle gray scale. Figure 9 Further, respectively, first reshaping (BHR) is performed on Q1, K1 and V, and second reshaping (FHR) is performed on Q2, K2 and V to obtain feature maps with a shape of CxBxHW / B, i.e. the first reshaped feature map including the third type of channel conversion feature map V, the first sub-type channel conversion feature map Q1 and the second sub-type channel conversion feature map K2, and the second reshaped feature map including the third type of channel conversion feature map V, the third sub-type channel conversion feature map Q2 and the fourth sub-type channel conversion feature map K2. Matrix multiplication (i.e. first fusion processing) is performed on Q1 and K1 to obtain a first fusion feature map, and similarly, matrix multiplication (i.e. second fusion processing) is performed on Q2 and K2 to obtain a second fusion feature map, as shown in the matrix image of the middle gray scale.

[0147] Then, the first attention map and the second attention map are pixel-wise multiplied to obtain an attention map. Since the channels in the dynamic range convolution are sorted by half, the attention map can be processed by channel equalization to obtain two feature maps with a shape of C / 2xHxW, i.e., a first type of channel attention map and a second type of channel attention map. Based on the reference sorting index transmitted in the dynamic range convolution process, the first type of channel attention map and the second type of channel attention map are restored to obtain a first restored attention map and a second restored attention map. Then, the first restored attention map and the second restored attention map are concatenated along the channel dimension, and the shape is adjusted to CxGxW through 1x1 point-wise convolution to obtain the final output target attention map, i.e., before the output of the final 1x1 point-wise convolution, the reordered features are arranged back to their original positions to maintain spatial consistency. Thus, the processing of the DHSA module in the image processing module ends.

[0148] Compared with the spatial attention with a fixed range or the attention along the channel dimension (depth-wise attention), on the one hand, high-resolution images can be processed only at the cost of great calculation ability and memory, and on the other hand, the self-attention is limited in a fixed range, which cannot obtain features in a larger range, resulting in a decline in the performance of image restoration.

[0149] In the embodiments of the present application, the degradation factors caused by severe weather cause the pixels of the image to display similar patterns. In order to assign different attention weights to clear pixels (background features) containing different pixel values and pixels of weather degradation factors (degradation pixels), by setting the number of feature element categories and the number of elements under each feature element category, different attention can be assigned between the feature elements (in the distribution map box) under each feature element category and each feature element category (distribution map box), a new distribution map self-attention is realized, which has a dynamic range of spatial attention, can capture visual obstacles caused by severe weather, and can effectively remove degradation factors in a global range to a certain extent. And its feature is obtained by sorting, which can give the self-attention a dynamic range of spatial receptive field, so that the convolution can extract the dependence relationship of the dynamic position of the weather-related degradation factors, thereby extracting the weather-related degradation factors in a spatial range, to enhance the aggregation of local and global features, and has the ability to effectively model the spatial dynamic range.

[0150] In a possible implementation, after obtaining the target attention map, the target attention map can be used to reconstruct the to-be-processed image to obtain a restored image corresponding to the to-be-processed image. That is, the output of the DHSA module can be used to determine the input of the DGFF module. Specifically, the processing procedure of the DGFF module can include: first performing third extraction processing on the image features of the target attention map to obtain a first attention feature map corresponding to the target attention map, and performing fourth extraction processing on the image features of the target attention map to obtain a second attention feature map corresponding to the target attention map, then performing activation processing on the first attention feature map according to the first attention feature map to obtain a gating feature map, and finally performing fusion processing on the gating feature map and the second attention feature map to obtain the restored image.

[0151] In the third extraction processing and the fourth extraction processing, the feature extraction scales are different, which can be understood as convolution processing with different kernel sizes, or can be understood as convolution processing of different types. For example, the third extraction processing can be 3x3 dilated convolution processing, and the fourth extraction processing can be 5x5 deep convolution processing, to capture multi-range and multi-scale information in the embedded feature matrix. Before the third extraction processing and the fourth extraction processing are performed on the image features of the target attention map, 1x1 point-by-point convolution processing can be performed first to increase the channel dimension by a factor of r, to increase the channel number of the target attention map by r times, where r can be 2, for example, and the present application does not limit this. For example, the shape of the target attention map is CxHxW, and through 1x1 point-by-point convolution operation, an expanded target attention map is obtained, which has a shape of 2CxHxW. Then, up-sampling processing, such as pixel shuffling operation, can be performed to expand the size of the expanded target attention map. Then, the third extraction processing and the fourth extraction processing are performed on the expanded target attention map.

[0152] Specifically, in the feature transformation process, the expanded target attention map can be guided into two parallel branches, and the third extraction processing and the fourth extraction processing are performed on the two branches respectively. That is, according to the multiple channel feature maps included in the target attention map, channel equalization processing is performed on the multiple channel feature maps included in the target attention map to obtain a first target class channel attention map and a second target class channel attention map. Here, the target attention map refers to the expanded target attention map after the up-sampling processing, and then the channel equalization processing is performed to obtain feature maps with halved channel numbers and unchanged spatial sizes, i.e., the first target class channel attention map and the second target class channel attention map.

[0153] Then, the image features of the first target class channel attention map are subjected to third extraction processing, such as 3x3 hole convolution processing, to obtain a first attention feature map, and the image features of the second target class channel attention map are subjected to fourth extraction processing, such as 5x5 deep convolution processing, to obtain a second attention feature map. The two branches are subjected to extraction processing respectively, which can enhance the extraction of multi-range and multi-scale information and capture multi-scale range information.

[0154] Further, according to the first attention feature map, the first attention feature map is subjected to activation processing to obtain a gated feature map. The activation processing refers to processing the first attention feature map through an activation function. The activation function is mainly used to map the input of a neuron to the output end. By introducing a nonlinear characteristic, the image processing model has the ability to learn and process complex and nonlinear problems. The gated feature map controls the transmission of information by introducing a gating mechanism, so that the model can selectively focus on important features or information while ignoring unimportant parts. The gated feature map can control the importance of different features through attention scores (i.e., attention maps), and the activation function is responsible for mapping these features to the output end.

[0155] Finally, the gated feature map and the second attention feature map are subjected to fusion processing to obtain a restored image. That is, the gated feature map and the second attention feature map are subjected to pixel-by-pixel multiplication to obtain the restored image. It should be noted that the restored image herein refers to the processing result of an image processing module, which can be used to determine the input of the next image processing module. Please refer to Figure 10 , Figure 10 is a schematic diagram of a DGFF module provided by an embodiment of the present application, Figure 10 Taking the DGFF module as a double-scale gated feedforward module (including two branches) as an example, the present application does not limit the multi-scale gated feedforward module (including multiple branches). The specific design can be based on the model structure and application scenario. For example, Figure 10 As shown in FIG. 5, the target attention map can be subjected to 1x1 point-by-point convolution processing first, which increases the channel number of the target attention map to r times, and then subjected to up-sampling processing to expand the size of the expanded target attention map. Further, the expanded target attention map is subjected to channel division processing to obtain two branches, i.e., a first target class channel attention map and a second target class channel attention map.

[0156] Further, one branch (the first target class channel attention map) is subjected to third extraction processing (3x3 hole convolution processing), such as Figure 10As shown in the right branch, the first attention feature map is obtained, and the fourth extraction processing (5x5 deep convolution processing) is performed on the second branch (the second target class channel attention map), as shown in the following formula 13: Figure 10 As shown in the left branch, the second attention feature map is obtained. Then, the first attention feature map is subjected to activation processing to obtain a gating feature map, and the gating feature map is fused with the second attention feature map, as shown in the following formula 14: Figure 10 As shown in the pixel-by-pixel multiplication, the output of the DGFF is obtained. Further, the above upsampling processing is restored, that is, the downsampling processing is performed, and the channel number of the output of the DGFF is adjusted to C by using the 1x1 point-by-point convolution, that is, the output with the shape of CxHxW is obtained, thereby the DGFF module integrates two different range and scale deep convolution paths in the transmission process. Specifically, the feature fusion process of the DGFF can be as shown in the following formula 12- formula 15:

[0157] F l,1 ,F l,2 =Split(Shuffle(Conv 1×1 (F l ))) formula 12

[0158]

[0159]

[0160] F l+1 =Conv 1×1 (Unshuffle(Mish(F l,2 )⊙F l,1 )) formula 15

[0161] In formula 12, F l,1 and F l,2 represent the first target class channel attention map and the second target class channel attention map, respectively, F l is the feature map input to the DGFF module, and is a tensor of Conv 1×1 represents the 1x1 point-by-point convolution, Shuffle represents the pixel shuffling, that is, the upsampling processing, and Split represents the channel splitting processing.

[0162] In formula 13, F l,1 on the left side of the equal sign represents the second attention feature map obtained by the fourth extraction processing, represents the 5x5 deep convolution, and F l,1 on the right side of the equal sign represents the second target class channel attention map without the fourth extraction processing.

[0163] In formula 14, F l,2denotes the first attention feature map obtained through the third extraction processing, denotes a 3x3 empty convolution, F l,2 denotes the first target class channel attention map without third extraction processing.

[0164] In formula 15, F l+1 denotes the output passed to the next stage, which is the output of the current image processing module and can be used to determine the input of the next image processing module. Conv 1×1 denotes a 1x1 pointwise convolution, and Unshuffle denotes pixel restoration, that is, corresponding down-sampling processing is performed on the up-sampling processing. Mish denotes a Mish activation function, which is used to represent an activation processing to obtain a gated feature map. ⊙ denotes a pixel-by-pixel multiplication, and F l,1 and F l,2 denote the second attention feature map and the first attention feature map, respectively.

[0165] It should be noted that the above process is an example of an image processing module. After each image processing module in the image processing model is processed, the restored image is obtained. For example Figure 4 In the formula, the output of the image processing module corresponding to L4 in the decoder is obtained after 3x3 convolution processing, and the final output image is obtained, that is, the restored image corresponding to the image to be processed.

[0166] It can be understood that using a single range or single scale convolution to enhance local context may ignore the correlation between degradation factors caused by dynamically distributed weather, because severe weather may cause target objects in the image to appear blurred, occluded, or color changed, and these problems may have different forms at different scales. Due to the limitation of the receptive field, a single scale convolution may not be able to capture these degradation factors at different scales at the same time, so it is easy to ignore the correlation between them. In this application, by expanding the size of the convolution kernel and using the empty mechanism, multi-scale information can be extracted, which enriches the representation of multi-range features and helps image restoration, so as to better cope with complex environmental factors such as weather changes. Expanding the size of the convolution kernel can increase the receptive field, so that the model can capture features in a larger range, and the empty mechanism can increase the receptive field without increasing the number of parameters, further improving the model's ability to extract multi-scale information, so as to capture multi-scale and multi-range information embedded in the feature matrix. As can be seen, the image processing model in this application can better understand and process degradation factors of different scales caused by weather changes, enhance its ability to effectively model visual features, and improve the robustness and performance of the model.

[0167] In the technical solutions provided in some embodiments of the present application, the image features of the to-be-processed image are extracted to obtain a feature map of the degraded image, then the feature elements included in the feature map are sorted according to the size relationship of the feature elements, to obtain a sorted feature map, then the sorted feature map is subjected to first feature recombination processing according to a preset number of feature element categories, to obtain a first recombined feature map, and the sorted feature map is subjected to second feature recombination processing according to a preset number of elements in each feature element category, to obtain a second recombined feature map; then an attention map is generated according to the first recombined feature map and the second recombined feature map, and the to-be-processed image is reconstructed according to the attention map, to obtain a restored image corresponding to the to-be-processed image. As can be seen, the pixels affected by the weather usually have similar pixel values, and the sorting processing can determine the pixels with similar pixel values and dispersed spatial positions, so that the clear pixels and the affected pixels are separated, which is conducive to extracting clear image features and degraded image features. By classifying the sorted pixels based on two different ways respectively, clear features and degraded features of different dimensions can be extracted respectively, an attention map focusing on preserving clear features and restoring degraded features can be generated, and thus the image restoration effect can be improved to a certain extent.

[0168] The embodiments of the present application further provide an image processing method. The image processing method described in the embodiments of the present application can be executed by an electronic device, which can be the image processing device 101 in the image processing system shown in Figure 1 The image processing device 101 in the image processing system shown in Figure 11 Figure 11 is another flow diagram of an image processing method provided by the embodiments of the present application. The image processing method includes the following steps S1101-S1104:

[0169] S1101, determining first loss data according to the difference between the restored image and the label image.

[0170] ​In the embodiments of the present application, in the process of training, the restored image refers to an image output by the image processing model, and is an image output by the image processing model to remove the degradation information caused by the bad weather. The restored image corresponds to a training image (to-be-processed image) in the training sample input to the initial image processing model. The label image is a clean image of the training image in the training sample, which can be used to refer to the image restoration effect of the restored image output by the initial image processing model. The training sample can be one training sample in the training data set received from another device, or can be obtained from an image platform related to the weather, for example, a training image selected from the rain and fog removal data set Outdoor-Rain, the raindrop data set RainDrop, the snow removal data set Snow100K (including a small snow data subset Snow100K-S and a large snow data subset Snow100K-L), and the like. The first loss data can be loss data (Loss) determined according to the L1 norm of the difference between the restored image and the label image. Specifically, the first loss data can be as shown in formula 16:

[0171]

[0172] In formula 16, indicates the first loss data, which can also be referred to as a reconstruct loss, I hq indicates a high-quality image (high quality), that is, the restored image after the above repair, I gt indicates a label image, that is, a reference image corresponding to the restored image. The L1 norm, also referred to as the sum of absolute errors (Sum Of Absolute Errors, SAE), is defined as the sum of the absolute values of each element in the vector. It calculates the absolute difference between the pixel value I hq (p) and the pixel value I gt (p) for each pixel p in the image, and then sums the differences of all pixel values in the entire image.

[0173] S1102, according to the correlation between each image block in the above restored image and each image block in the above label image, determine the second loss data.

[0174] In the embodiments of the present application, since Only the pixel-level similarity between the restored image and the reference image is adjusted, while the linear correlation at the image block level is ignored. Therefore, a loss data can be constructed to determine the correlation between each image block in the restored image and each image block in the label image, so as to enhance the overall linear correlation between the restored image and the label image (clean image), that is, the second loss function. For example, assuming an image of a clean background and an image containing information affected by snow, if the pixel value size relationship within the image is sorted, it will be found that the sorting indexes are different, because the information affected by snow changes the darker part to the brighter part, so the sorting index of this part may change, that is, the inherent relationship of the pixel value within the image is broken by the similar mode of the degradation factor caused by the weather.

[0175] However, learning the pixel value relationship in the label image (clean image) can require the sorting indexes of the pixel sequences in the restored image and the label image output by the image processing model to remain consistent, that is, to force the degraded pixels to occupy their original positions according to the original pixel value sorting. Among them, the correlation at the image block level can be determined by determining the Pearson correlation coefficient between the images, that is, to ensure the reconstruction of the linear relationship between the restored image and the clean image. Specifically, the first pixel value mean and the first pixel value standard deviation of the restored image can be determined according to the pixel values of each pixel point in the restored image, and the second pixel value mean and the second pixel value standard deviation of the label image can be determined according to the pixel values of each pixel point in the label image, and then the Pearson correlation coefficient can be calculated according to the first pixel value mean, the first pixel value standard deviation, the second pixel value mean and the second pixel value standard deviation, and the second loss data can be determined according to the Pearson correlation coefficient.

[0176] Among them, the first pixel value mean is the mean of the pixel values of each pixel point in the restored image, the first pixel value standard deviation is the standard deviation of the pixel values of each pixel point in the restored image, the second pixel value mean is the mean of the pixel values of each pixel point in the label image, and the second pixel value standard deviation is the standard deviation of the pixel values of each pixel point in the label image. Since the Pearson correlation coefficient is used to measure the statistical quantity of the linear correlation between two variables X and Y. Its value is between -1 and 1, when the correlation coefficient is 1, it means that the two variables are positively correlated; when the correlation coefficient is -1, it means that the two variables are negatively correlated; when the correlation coefficient is 0, it means that the two variables have no linear relationship. Therefore, the Pearson correlation coefficient can be used as a means to adjust the linear relationship, specifically, the Pearson correlation coefficient can be calculated as shown in formula 17:

[0177]

[0178] In formula 17, ρ(I hq ,Igt represents the Pearson correlation coefficient, 3HW represents the pixel value of the 3-channel (red, green, blue) pixel, and i represents the serial number of the pixel, represents the pixel value of the i-th pixel in the recovered image, represents the pixel value of the i-th pixel in the label image, and σ(I {·} ) represent the mean and standard deviation of the image sequence, as and respectively represent the pixel value mean of the recovered image and the label image, i.e. the first pixel value mean and the second pixel value mean described above, σ(I hq ) and σ(I gt ) respectively represent the pixel value standard value of the recovered image and the label image, i.e. the first pixel value standard deviation and the second pixel value standard deviation described above. The value of ρ(I hq ,I gt ) is in the range of [-1, 1]. It can be understood that when the recovered image I hq and the label image I gt are completely correlated, the value of the function ρ(I hq ,I gt ) reaches 1, and in the case of negative correlation, the value is -1.

[0179] Further, the second loss data, which can also be referred to as the correlation loss data, can be determined according to the Pearson correlation coefficient. Since the value of the Pearson correlation coefficient is between [-1, 1], a loss data can be designed to have a value between [0, 1], and specifically, the second loss data can be as shown in equation 18:

[0180]

[0181] In equation 18, represents the second loss data, which can also be referred to as the correlation loss data, and ρ(I hq ,I gt ) represents the Pearson correlation coefficient, and by the range of the second loss data can be converted to [0, 1]. It can be understood that when the recovered image and the label image are completely the same (aligned), the value of the second loss data is 0.

[0182]

[0183] S1103, determine the target loss data according to the sum of the first loss data and the second loss data.

[0184] In the embodiments of the present application, the first loss data is determined based on the difference between the restored image and the label image, and the second loss data is determined based on the correlation between each image block in the restored image and each image block in the label image. Specifically, the target loss data can be as shown in formula 19:

[0185]

[0186] In formula 19, denotes the target loss data, is the first loss data, is the second loss data, and a is the weight of the second loss data. The value of a can be 1 or other values, such as a value less than 1. Optionally, the first loss data can also correspond to a weight, and the present application does not limit the weight of the first loss data and the second loss data.

[0187] S1104, training the initial image processing model based on the target loss data to obtain a target image processing model.

[0188] In the embodiments of the present application, the image processing device can adjust the model parameters of the initial image processing model, such as the model parameters in each image processing module, specifically including the model parameters in the DHSA module, according to the target loss data to obtain the target image processing model.

[0189] Optionally, the image processing device can adjust the model parameters of the initial image processing model by using the gradient descent method. When the gradient descent method is used to update the model parameters, the gradient of the loss data, such as the loss function, is calculated, and then the model parameters are iteratively updated according to the gradient to gradually converge the initial image processing model to improve the performance of the model in image restoration of the image affected by the bad weather.

[0190] In the technical solutions provided in some embodiments of the present application, by extracting image features of a to-be-processed image, a feature map of a degraded image is obtained, and then the feature elements included in the feature map are sorted according to the size relationship of the feature elements, to obtain a sorted feature map, and then the sorted feature map is subjected to first feature reorganization processing according to a preset number of feature element categories, to obtain a first reorganized feature map, and the sorted feature map is subjected to second feature reorganization processing according to a preset number of elements in each feature element category, to obtain a second reorganized feature map; then an attention map is generated according to the first reorganized feature map and the second reorganized feature map, and the to-be-processed image is reconstructed according to the attention map, to obtain a restored image corresponding to the to-be-processed image. As can be seen, since the pixels affected by the weather are usually close in pixel value, the sorting processing can determine the pixels close in pixel value and dispersed in space, so that the clear pixels and the affected pixels are separated, which is conducive to extracting clear image features and degraded image features. By classifying the sorted pixels based on two different ways respectively, clear features and degraded features of different dimensions can be extracted respectively, an attention map focusing on preserving clear features and restoring degraded features can be generated, so that the image restoration effect can be improved to a certain extent.

[0191] The image restoration effect of the image processing method provided in the embodiments of the present application is quantitatively and qualitatively evaluated as follows.

[0192] Please refer to Figure 12 , Figure 12 is an evaluation effect diagram of the image processing method provided in the embodiments of the present application. As Figure 12 indicated, the image processing model provided in the embodiments of the present application and other models for image restoration processing of degraded images caused by severe weather are respectively used to perform image restoration processing on the degraded images in the synthetic data set and the real data set affected by severe weather. The data set can include a snow removal data set Snow100K (including a small snow data subset Snow100K-S and a large snow data subset Snow100K-L), a rain and fog removal data set Outdoor-Rain, and a raindrop removal data set RainDrop. The above-mentioned other models for image restoration processing can be, for example, MPR-Net model, MAXIM model, Restormer model, and integrated models such as WGWS-Net.

[0193] As Figure 12As shown, the upper half of the table shows the processing results based on specific tasks, that is, quantitative comparison of image restoration processing for degraded images affected by a specific type of severe weather, and the lower half of the table shows the processing results of the integrated model. In order to perform quantitative evaluation, the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) indicators can be used to comprehensively compare and analyze the restored images. PSNR is an objective standard for measuring image distortion or noise level, which is defined based on the Mean-Square Error (MSE), and is used to calculate the similarity between the original image and the processed image. The larger the PSNR value, the more similar the images are, and the better the quality. SSIM is an index for measuring the similarity of two images, which compares the similarity of images in terms of brightness, contrast and structure. The value of SSIM is usually between -1 and 1, and the larger the value, the more similar the images are. As can be seen, the processing results of the image processing model proposed in the present application are the best results, that is, the values of PSNR and SSIM are the largest, as shown in the bold content of the middle table, which has a more significant performance advantage compared to other models. Figure 12

[0194] In order to perform qualitative evaluation, degraded images affected by different severe weather are also processed and visually compared. For example, the same degraded image can be processed by the image processing model of the present application and the existing model for image restoration processing of degraded images caused by severe weather to obtain restored images for visual comparison. Please refer to Figure 13 , Figure 13 is a comparison effect diagram of the restored image provided by the embodiment of the present application, as Figure 13 The comparison is made in snow scene, rain and fog scene, and raindrop scene. In the snow scene, the first image is the input snow image, which includes many degradation factors affected by snow, and can be a test image obtained from the Snow-100K dataset. The second image is the restored image obtained by the existing model, the third image is the restored image obtained by the image processing model of the present application, and the fourth image is the reference clean image. As can be seen, by comparing the third image with other images, the image processing model in the present application can better process small and large snow spots, etc., so that the restored image has better effect.

[0195] ​Similarly, in the rain and fog scene, the first image is the input rain and fog image, which includes more degradation factors affected by rain and fog, and can be a test image obtained from the Outdoor-Rain dataset. The second image is the restored image obtained by processing the existing model, the third image is the restored image obtained by processing the image processing model provided in the embodiments of the present application, and the fourth image is the reference clean image. In the raindrop scene, the first image is the input raindrop image, which includes more degradation factors affected by raindrops, and can be a test image obtained from the RainDrop dataset. The second image is the restored image obtained by processing the existing model, the third image is the restored image obtained by processing the image processing model provided in the embodiments of the present application, and the fourth image is the reference clean image.

[0196] Please refer to Figure 14 , Figure 14 is another comparison effect diagram of restoring an image provided by the embodiments of the present application, as Figure 14 shown, taking the image restoration effect in three real weather removal cases (real dataset) as an example for visual comparison. The leftmost column of images is the image to be processed, the middle column of images is the restored image obtained by processing the existing model, and the rightmost column of images is the restored image obtained by processing the image processing model provided in the embodiments of the present application. As can be seen, the image processing model provided in the embodiments of the present application has certain superiority in removing weather degradation factors from degraded images affected by various different severe weather, improves the image restoration effect, especially when it comes to restoring challenging weather conditions, such as complex haze and rain marks, to a certain extent, it can present a visual effect that meets the user's needs, and achieves good performance on various datasets, proving its effectiveness in restoring real-world images.

[0197] In order to further demonstrate the practical applicability of the image processing method provided in the embodiments of the present application in actual severe weather removal, and its ability to improve its performance in downstream detection tasks. Please refer to Figure 15 and Figure 16 , Figure 15 and Figure 16 are an effect comparison diagram for executing a downstream task provided by the embodiments of the present application, Figure 15 and Figure 16 take the restored images corresponding to two real-world images and the execution of the target detection task in an application programming interface (Application Programming Interface, API) as an example for explanation. Figure 15 and Figure 16 The images above all show the results of detecting degraded images affected by severe weather, Figure 15 and Figure 16The images below all show the results of detecting the restored image obtained based on the image processing model proposed in the present application. As can be seen, the embodiments of the present application can effectively eliminate the snowflakes in the image and retain the clear background image. Compared with the results of target detection based on the degraded image, target detection based on the restored image can make the detector identify the missing door and building, so that the performance of the downstream task is improved, thereby enhancing the effectiveness of the downstream detection application.

[0198] The above describes the method of the embodiments of the present application in detail. In order to facilitate better implementation of the above scheme of the embodiments of the present application, correspondingly, the device of the embodiments of the present application is provided below.

[0199] Please refer to Figure 17 , Figure 17 is a structural schematic diagram of an image processing device provided by the embodiments of the present application. The image processing device 170 can be used to perform the corresponding steps in the image processing method shown in Figure 2 and Figure 11 . The image processing device 170 includes the following units:

[0200] The extraction unit 1701 is configured to extract the image features of the to-be-processed image to obtain a feature map corresponding to the to-be-processed image.

[0201] The sorting unit 1702 is configured to sort the feature elements included in the feature map according to the size relationship of the feature elements to obtain a sorted feature map.

[0202] The processing unit 1703 is configured to perform first feature reorganization processing on the sorted feature map according to a preset number of feature element categories to obtain a first reorganized feature map, and perform second feature reorganization processing on the sorted feature map according to a preset number of elements under each feature element category to obtain a second reorganized feature map.

[0203] The generation unit 1704 is configured to generate an attention map according to the first reorganized feature map and the second reorganized feature map, and perform reconstruction processing on the to-be-processed image according to the attention map to obtain a restored image corresponding to the to-be-processed image.

[0204] In a possible implementation manner, the extraction unit 1701 is configured to extract the image features of the to-be-processed image to obtain a feature map corresponding to the to-be-processed image, and specifically configured to:

[0205] perform first extraction processing on the image features of the to-be-processed image to obtain an initial feature map corresponding to the to-be-processed image;

[0206] According to a size relationship of each feature element included in the initial feature map, the feature elements included in the initial feature map are sorted in a horizontal direction and a vertical direction to obtain an initial sorted feature map;

[0207] An image feature of the initial sorted feature map is subjected to a second extraction processing to obtain the feature map.

[0208] In a possible implementation, the initial feature map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; the extraction unit 1701 is configured to sort the feature elements included in the initial feature map in a horizontal direction and a vertical direction according to a size relationship of each feature element included in the initial feature map to obtain an initial sorted feature map, and specifically configured to:

[0209] According to the plurality of channel feature maps included in the initial feature map, the plurality of channel feature maps are subjected to channel equalization processing to obtain a first type of channel feature map and a second type of channel feature map;

[0210] According to a size relationship of each feature element included in the first type of channel feature map, the feature elements in each channel feature map included in the first type of channel feature map are sorted in the horizontal direction and the vertical direction to obtain a first type of channel sorted feature map;

[0211] The first type of channel sorted feature map and the second type of channel feature map are subjected to concatenation processing of corresponding channels to obtain the initial sorted feature map.

[0212] In a possible implementation, the attention map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; the generation unit 1704 is configured to reconstruct the to-be-processed image according to the attention map to obtain a recovered image corresponding to the to-be-processed image, and specifically configured to:

[0213] According to the plurality of channel feature maps included in the attention map, the plurality of channel feature maps included in the attention map are subjected to channel equalization processing to obtain a first type of channel attention map and a second type of channel attention map;

[0214] The first type of channel attention map and the second type of channel attention map are subjected to recovery processing according to a reference sorting index, to obtain a first recovered attention map and a second recovered attention map, the reference sorting index being used to indicate a position of each feature element included in the first type of channel sorted feature map in the first type of channel feature map;

[0215] The first recovered attention map and the second recovered attention map are subjected to concatenation processing of corresponding channels to obtain a target attention map;

[0216] The target attention map is used for reconstruction processing on the to-be-processed image, to obtain a restored image corresponding to the to-be-processed image.

[0217] In a possible implementation, the generation unit 1704 is configured to perform reconstruction processing on the to-be-processed image according to the target attention map, to obtain a restored image corresponding to the to-be-processed image, and specifically configured to:

[0218] perform third extraction processing on image features of the target attention map to obtain a first attention feature map corresponding to the target attention map, and perform fourth extraction processing on the image features of the target attention map to obtain a second attention feature map corresponding to the target attention map, wherein the feature extraction scales of the third extraction processing and the fourth extraction processing are different;

[0219] perform activation processing on the first attention feature map according to the first attention feature map, to obtain a gating feature map;

[0220] perform fusion processing on the gating feature map and the second attention feature map, to obtain the restored image.

[0221] In a possible implementation, the target attention map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; and the generation unit 1704 is configured to perform third extraction processing on image features of the target attention map to obtain a first attention feature map corresponding to the target attention map, and perform fourth extraction processing on the image features of the target attention map to obtain a second attention feature map corresponding to the target attention map, and specifically configured to:

[0222] perform channel equalization processing on a plurality of channel feature maps included in the target attention map according to the plurality of channel feature maps included in the target attention map, to obtain a first target class channel attention map and a second target class channel attention map;

[0223] perform the third extraction processing on image features of the first target class channel attention map to obtain the first attention feature map, and perform the fourth extraction processing on image features of the second target class channel attention map to obtain the second attention feature map.

[0224] In a possible implementation, the feature map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; and the sorting unit 1702 is configured to perform sorting processing on feature elements included in the feature map according to a size relationship of the feature elements included in the feature map, to obtain a sorted feature map, and specifically configured to:

[0225] perform channel equalization processing on a plurality of channel feature maps included in the feature map according to the plurality of channel feature maps included in the feature map, to obtain a plurality of class channel feature maps.

[0226] According to a size relationship of each feature element included in the multi-class channel feature map, the feature elements included in the multi-class channel feature map are respectively sorted to obtain a sorted feature map corresponding to each of the multi-class channel feature maps.

[0227] In a possible implementation, the sorting unit 1702 is configured to sort the feature elements included in each of the multi-class channel feature maps according to a size relationship of each feature element included in the multi-class channel feature map, to obtain a sorted feature map corresponding to each of the multi-class channel feature maps, and specifically configured to:

[0228] The feature elements in the plurality of channel feature maps included in each of the multi-class channel feature maps are subjected to feature reorganization processing, so as to convert the multi-class channel feature map from a three-dimensional tensor to a two-dimensional tensor, to obtain a channel conversion feature map corresponding to each of the multi-class channel feature maps;

[0229] According to a size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorted feature map corresponding to each of the multi-class channel feature maps.

[0230] In a possible implementation, the sorting unit 1702 is configured to sort the channel conversion feature map corresponding to each of the multi-class channel feature maps according to a size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, and specifically configured to:

[0231] According to a size relationship of each feature element included in a specified class channel conversion feature map in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the feature elements in each channel feature map included in the specified class channel conversion feature map are sorted to obtain a specified class channel sorted feature map and a sorting index, where the sorting index is used to indicate a position of each feature element included in the specified class channel sorted feature map in the specified class channel conversion feature map.

[0232] According to the sorting index, the channel conversion feature maps other than the specified class channel conversion feature map in the channel conversion feature map corresponding to each of the multi-class channel feature maps are sorted.

[0233] In a possible implementation, the multiple types of channel feature maps include a third type of channel feature map, a fourth type of channel feature map, and a fifth type of channel feature map; the processing unit 1703 is configured to perform first feature reorganization processing on the sorted feature map according to a preset number of feature element categories to obtain a first reorganized feature map, and perform second feature reorganization processing on the sorted feature map according to an element quantity in each category to obtain a second reorganized feature map, and specifically configured to:

[0234] According to the multiple channel feature maps included in the fourth type of channel sorted feature map, the multiple channel feature maps included in the fourth type of channel sorted feature map are subjected to channel equalization processing to obtain a first subcategory channel sorted feature map and a second subcategory channel sorted feature map.

[0235] According to the multiple channel feature maps included in the fifth type of channel sorted feature map, the multiple channel feature maps included in the fifth type of channel sorted feature map are subjected to channel equalization processing to obtain a third subcategory channel sorted feature map and a fourth subcategory channel sorted feature map.

[0236] According to the number of feature element categories, the third type of channel sorted feature map, the first subcategory channel sorted feature map, and the second subcategory channel sorted feature map are subjected to first feature reorganization processing to obtain the first reorganized feature map.

[0237] According to the element quantity, the third type of channel sorted feature map, the third subcategory channel sorted feature map, and the fourth subcategory channel sorted feature map are subjected to second feature reorganization processing to obtain the second reorganized feature map.

[0238] The element quantity in each feature element category corresponding to the first reorganized feature map is greater than the element quantity in each feature element category corresponding to the second reorganized feature map.

[0239] In a possible implementation, the processing unit 1703 is configured to perform first feature reorganization processing on the third type of channel sorted feature map, the first subcategory channel sorted feature map, and the second subcategory channel sorted feature map according to the number of feature element categories to obtain the first reorganized feature map, and specifically configured to:

[0240] According to the number of feature element categories, the third type of channel sorted feature map, the first subcategory channel sorted feature map, and the second subcategory channel sorted feature map respectively include feature elements in each channel feature map, and the third type of channel sorted feature map, the first subcategory channel sorted feature map, and the second subcategory channel sorted feature map are respectively converted from two-dimensional tensors to three-dimensional tensors to obtain a third type of channel converted feature map, a first subcategory channel converted feature map, and a second subcategory channel converted feature map.

[0241] generate the first reorganized feature map according to the third-class channel conversion feature map, the first-sub-class channel conversion feature map and the second-sub-class channel conversion feature map;

[0242] The height parameter of the three-dimensional tensor is the element category quantity, and the width parameter of the three-dimensional tensor is the element quantity in each element category determined according to the element quantity in each channel feature map and the element category quantity.

[0243] In a possible implementation, the processing unit 1703 is configured to perform second feature reorganization processing on the third-class channel sorting feature map, the third-sub-class channel sorting feature map and the fourth-sub-class channel sorting feature map according to the element quantity, to obtain the second reorganized feature map, and specifically configured to:

[0244] perform feature reorganization processing on the feature elements in each channel feature map included in the third-class channel sorting feature map, the third-sub-class channel sorting feature map and the fourth-sub-class channel sorting feature map according to the element quantity, to convert the third-class channel sorting feature map, the third-sub-class channel sorting feature map and the fourth-sub-class channel sorting feature map from two-dimensional tensors into three-dimensional tensors respectively, to obtain a third-class channel conversion feature map, a third-sub-class channel conversion feature map and a fourth-sub-class channel conversion feature map;

[0245] generate the second reorganized feature map according to the third-class channel conversion feature map, the third-sub-class channel conversion feature map and the fourth-sub-class channel conversion feature map;

[0246] The height parameter of the three-dimensional tensor is the element quantity, and the width parameter of the three-dimensional tensor is an element category quantity determined according to the element quantity in each channel feature map and the element quantity.

[0247] In a possible implementation, the multi-class channel feature map includes a third-class channel feature map, a fourth-class channel feature map and a fifth-class channel feature map; the first reorganized feature map includes a third-class channel conversion feature map corresponding to the third-class channel feature map, and a first-sub-class channel conversion feature map and a second-sub-class channel conversion feature map corresponding to the fourth-class channel feature map; and the second reorganized feature map includes a third-class channel conversion feature map corresponding to the third-class channel feature map, and a third-sub-class channel conversion feature map and a fourth-sub-class channel conversion feature map corresponding to the fifth-class channel feature map.

[0248] The generating unit 1704 is configured to generate an attention map according to the first reorganized feature map and the second reorganized feature map, and specifically configured to:

[0249] generate a first attention map according to the third type channel conversion feature map, the first sub-type channel conversion feature map and the second sub-type channel conversion feature map, and generate a second attention map according to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map;

[0250] perform fusion processing on the first attention map and the second attention map to obtain the attention map.

[0251] In a possible implementation, the generation unit 1704 is configured to generate a first attention map according to the third type channel conversion feature map, the first sub-type channel conversion feature map and the second sub-type channel conversion feature map, and generate a second attention map according to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map, and specifically configured to:

[0252] perform first fusion processing on the first sub-type channel conversion feature map and the second sub-type channel conversion feature map to obtain a first fusion feature map, and perform second fusion processing on the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map to obtain a second fusion feature map;

[0253] perform first normalization processing on the first fusion feature map and the third type channel conversion feature map to obtain the first attention map, and perform second normalization processing on the second fusion feature map and the third type channel conversion feature map to obtain the second attention map.

[0254] In a possible implementation, the image to be processed is a training image in a training sample, and the training sample further includes a label image corresponding to the image to be processed; and the image processing apparatus 170 further includes:

[0255] The determination unit 1705 is configured to determine first loss data according to a difference between the restored image and the label image, determine second loss data according to a correlation between each image block in the restored image and each image block in the label image, and determine target loss data according to a sum of the first loss data and the second loss data.

[0256] The training unit 1706 is configured to train an initial image processing model based on the target loss data to obtain a target image processing model.

[0257] In a possible implementation, the determination unit 1705 is configured to determine second loss data according to a correlation between each image block in the restored image and each image block in the label image, and specifically configured to:

[0258] According to pixel values of each pixel point in the recovery image, a first pixel value mean and a first pixel value standard deviation of the recovery image are determined, and according to pixel values of each pixel point in the label image, a second pixel value mean and a second pixel value standard deviation of the label image are determined.

[0259] According to the first pixel value mean, the first pixel value standard deviation, the second pixel value mean and the second pixel value standard deviation, a Pearson correlation coefficient is calculated, and the second loss data is determined according to the Pearson correlation coefficient.

[0260] According to an embodiment of the present application, Figure 2 and Figure 11 The steps involved in the method shown in the figure can be executed by Figure 17 each unit in the image processing apparatus shown in the figure. For example, Figure 2 Step S201 shown in the figure is executed by Figure 17 the extraction unit 1701, step S202 is executed by Figure 17 the sorting unit 1702, step S203 is executed by Figure 17 the processing unit 1703, and step S204 is executed by Figure 17 the generation unit 1704. For another example, Figure 11 Steps S1101-S1103 shown in the figure are all executed by Figure 17 the determination unit 1705, Figure 11 Step S1104 shown in the figure is executed by Figure 17 the training unit 1706.

[0261] According to an embodiment of the present application, Figure 17 Each unit in the image processing apparatus 170 shown in the figure can be combined into one or several other units respectively or all, or some of the units can be further split into a plurality of units with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions, and in actual application, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the image processing apparatus 170 can also include other units, and in actual application, these functions can also be realized by other units, and can be realized by multiple units. According to another embodiment of the present application, the image processing apparatus 170 can be realized by running the program capable of executing the above-mentioned steps on a general computing device including processing elements and storage elements such as central processing unit (CPU), random access memory (RAM), read-only memory (ROM) and the like, and the like. Figure 2 and Figure 10a computer program (including program codes) for implementing the steps of the corresponding method shown in the above embodiments of the image processing method. Figure 17 the image processing device 170 shown in the above embodiments of the image processing method, and the image processing method implemented by the embodiments of the present application. The computer program can be recorded on a computer readable storage medium, for example, and loaded into the image processing device 180 through the computer readable storage medium. Figure 1 the image processing device 101 of the image processing system shown in the above embodiments of the image processing method, and runs in the image processing device 101.

[0262] Based on the description of the above embodiments of the image processing method, the embodiments of the present application further disclose an image processing device, please refer to Figure 18 The image processing device 180 can at least include a processor 1801, an input device 1802, an output device 1803 and a memory 1804. Wherein, the processor 1801, the input device 1802, the output device 1803 and the memory 1804 in the image processing device 180 can be connected through a bus or other means.

[0263] The above-mentioned memory 1804 is a memory device in the image processing device 180, used for storing programs and data. It can be understood that the memory 1804 here can include the built-in storage medium of the image processing device, and of course can also include the expansion storage medium supported by the image processing device 180. The memory 1804 provides a storage space which stores the operating system of the image processing device 180. And, the computer program (including program codes) is also stored in the storage space. It should be noted that the computer storage medium here can be a high-speed RAM memory; optionally, it can also be at least one computer storage medium away from the above-mentioned processor, and the above-mentioned processor can be called a central processing unit (CPU), which is the core and control center of the image processing device, used for running the computer program stored in the above-mentioned memory 1804.

[0264] In an embodiment, the computer program stored in the memory 1804 can be loaded and executed by the processor 1801 to realize the corresponding steps of the method in the above-mentioned embodiments of the image processing method; specifically, the processor 1801 loads and executes the computer program stored in the memory 1804, which is used for:

[0265] extracting image features of the image to be processed to obtain a feature map corresponding to the image to be processed;

[0266] According to the size relationship of each feature element included in the feature map, the feature elements included in the feature map are sorted to obtain a sorted feature map;

[0267] perform first feature reorganization processing on the sorted feature map according to a preset number of feature element categories, to obtain a first reorganized feature map, and perform second feature reorganization processing on the sorted feature map according to a preset number of elements in each feature element category, to obtain a second reorganized feature map;

[0268] generate an attention map according to the first reorganized feature map and the second reorganized feature map, and perform reconstruction processing on the to-be-processed image according to the attention map, to obtain a restored image corresponding to the to-be-processed image.

[0269] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to extract image features of a to-be-processed image, to obtain a feature map corresponding to the to-be-processed image, and specifically to:

[0270] perform first extraction processing on the image features of the to-be-processed image, to obtain an initial feature map corresponding to the to-be-processed image;

[0271] perform sorting processing on feature elements included in the initial feature map in a horizontal direction and a vertical direction according to a size relationship of the feature elements, to obtain an initial sorted feature map;

[0272] perform second extraction processing on image features of the initial sorted feature map, to obtain the feature map.

[0273] In a possible implementation, the initial feature map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; the processor 1801 loads and executes a computer program stored in the memory 1804, to perform sorting processing on feature elements included in the initial feature map in a horizontal direction and a vertical direction according to a size relationship of the feature elements, to obtain an initial sorted feature map, and specifically to:

[0274] perform channel equalization processing on the plurality of channel feature maps according to the plurality of channel feature maps included in the initial feature map, to obtain a first type of channel feature map and a second type of channel feature map;

[0275] perform sorting processing on feature elements in each channel feature map included in the first type of channel feature map in the horizontal direction and the vertical direction according to a size relationship of the feature elements included in the first type of channel feature map, to obtain a first type of channel sorted feature map;

[0276] perform series connection processing on the first type of channel sorted feature map and the second type of channel feature map in corresponding channels, to obtain the initial sorted feature map.

[0277] In a possible implementation, the attention map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; the processor 1801 loads and executes a computer program stored in the memory 1804, to perform reconstruction processing on the to-be-processed image according to the attention map, to obtain a restored image corresponding to the to-be-processed image, and specifically for:

[0278] According to the plurality of channel feature maps included in the attention map, the plurality of channel feature maps included in the attention map are subjected to channel equalization processing, to obtain a first type of channel attention map and a second type of channel attention map;

[0279] According to the reference order index, the first type of channel attention map and the second type of channel attention map are subjected to restoration processing, to obtain a first restored attention map and a second restored attention map, the reference order index being used to indicate the element position of each feature element included in the first type of channel ordering feature map in the first type of channel feature map;

[0280] The first restored attention map and the second restored attention map are subjected to corresponding channel concatenation processing, to obtain a target attention map;

[0281] According to the target attention map, the to-be-processed image is subjected to reconstruction processing, to obtain a restored image corresponding to the to-be-processed image.

[0282] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to perform reconstruction processing on the to-be-processed image according to the target attention map, to obtain a restored image corresponding to the to-be-processed image, and specifically for:

[0283] The image features of the target attention map are subjected to third extraction processing, to obtain a first attention feature map corresponding to the target attention map, and the image features of the target attention map are subjected to fourth extraction processing, to obtain a second attention feature map corresponding to the target attention map, wherein the feature extraction scales of the third extraction processing and the fourth extraction processing are different;

[0284] According to the first attention feature map, the first attention feature map is subjected to activation processing, to obtain a gating feature map;

[0285] The gating feature map and the second attention feature map are subjected to fusion processing, to obtain the restored image.

[0286] In a possible implementation, the target attention map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; the processor 1801 loads and executes the computer program stored in the memory 1804, to perform third extraction processing on image features of the target attention map to obtain a first attention feature map corresponding to the target attention map, and perform fourth extraction processing on the image features of the target attention map to obtain a second attention feature map corresponding to the target attention map, specifically for:

[0287] According to the plurality of channel feature maps included in the target attention map, the plurality of channel feature maps included in the target attention map are subjected to channel equalization processing to obtain a first target class channel attention map and a second target class channel attention map;

[0288] The image features of the first target class channel attention map are subjected to the third extraction processing to obtain the first attention feature map, and the image features of the second target class channel attention map are subjected to the fourth extraction processing to obtain the second attention feature map.

[0289] In a possible implementation, the feature map includes a plurality of channel feature maps corresponding to a plurality of channels respectively; the processor 1801 loads and executes the computer program stored in the memory 1804, to perform sorting processing on feature elements included in the feature map according to a size relationship of the feature elements included in the feature map to obtain a sorted feature map, specifically for:

[0290] According to the plurality of channel feature maps included in the feature map, the plurality of channel feature maps are subjected to channel equalization processing to obtain a plurality of class channel feature maps;

[0291] According to a size relationship of feature elements included in the plurality of class channel feature maps, the feature elements included in the plurality of class channel feature maps are subjected to sorting processing respectively to obtain sorted feature maps corresponding to the plurality of class channel feature maps respectively.

[0292] In a possible implementation, the processor 1801 loads and executes the computer program stored in the memory 1804, to perform sorting processing on feature elements included in the plurality of class channel feature maps respectively according to a size relationship of the feature elements included in the plurality of class channel feature maps to obtain sorted feature maps corresponding to the plurality of class channel feature maps respectively, specifically for:

[0293] The feature elements in a plurality of channel feature maps included in the plurality of class channel feature maps are subjected to feature recombination processing to convert the plurality of class channel feature maps from three-dimensional tensors to two-dimensional tensors respectively to obtain channel conversion feature maps corresponding to the plurality of class channel feature maps respectively;

[0294] According to a size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorting feature map corresponding to each of the multi-class channel feature maps.

[0295] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to perform sorting processing on the channel conversion feature map corresponding to each of the multi-class channel feature maps according to a size relationship of each feature element included in the channel conversion feature map, and specifically to:

[0296] According to a size relationship of each feature element included in the specified-class channel conversion feature map, feature elements in each channel feature map included in the specified-class channel conversion feature map are sorted to obtain a specified-class channel sorting feature map and a sorting index, where the sorting index is used to indicate a position of each feature element included in the specified-class channel sorting feature map in the specified-class channel conversion feature map.

[0297] According to the sorting index, the channel conversion feature map other than the specified-class channel conversion feature map among the channel conversion feature maps corresponding to each of the multi-class channel feature maps is sorted.

[0298] In a possible implementation, the multi-class channel feature maps include a third-class channel feature map, a fourth-class channel feature map, and a fifth-class channel feature map; the processor 1801 loads and executes a computer program stored in the memory 1804, to perform first feature recombination processing on the sorting feature map according to a preset number of feature element classes to obtain a first recombination feature map, and perform second feature recombination processing on the sorting feature map according to a number of elements under each feature element class to obtain a second recombination feature map, and specifically to:

[0299] According to the fourth-class channel sorting feature map, channel equalization processing is performed on the plurality of channel feature maps included in the fourth-class channel sorting feature map to obtain a first sub-class channel sorting feature map and a second sub-class channel sorting feature map.

[0300] According to the fifth-class channel sorting feature map, channel equalization processing is performed on the plurality of channel feature maps included in the fifth-class channel sorting feature map to obtain a third sub-class channel sorting feature map and a fourth sub-class channel sorting feature map.

[0301] perform first feature reorganization processing on the third-type channel-ordered feature map, the first-subtype channel-ordered feature map, and the second-subtype channel-ordered feature map according to the element category quantity, to obtain the first reorganized feature map;

[0302] perform second feature reorganization processing on the third-type channel-ordered feature map, the third-subtype channel-ordered feature map, and the fourth-subtype channel-ordered feature map according to the element quantity, to obtain the second reorganized feature map;

[0303] The element quantity of each feature element category corresponding to the first reorganized feature map is greater than the element quantity of each feature element category corresponding to the second reorganized feature map.

[0304] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to perform first feature reorganization processing on the third-type channel-ordered feature map, the first-subtype channel-ordered feature map, and the second-subtype channel-ordered feature map according to the element category quantity, to obtain the first reorganized feature map, and specifically to:

[0305] perform feature reorganization processing on feature elements in each channel feature map included in the third-type channel-ordered feature map, the first-subtype channel-ordered feature map, and the second-subtype channel-ordered feature map according to the element category quantity, to convert the third-type channel-ordered feature map, the first-subtype channel-ordered feature map, and the second-subtype channel-ordered feature map from two-dimensional tensors into three-dimensional tensors, to obtain a third-type channel-converted feature map, a first-subtype channel-converted feature map, and a second-subtype channel-converted feature map;

[0306] generate the first reorganized feature map according to the third-type channel-converted feature map, the first-subtype channel-converted feature map, and the second-subtype channel-converted feature map;

[0307] The height parameter of the three-dimensional tensor is the element category quantity, and the width parameter of the three-dimensional tensor is an element quantity in each feature element category determined according to the element quantity in the channel feature map and the element category quantity.

[0308] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to perform second feature reorganization processing on the third-type channel-ordered feature map, the third-subtype channel-ordered feature map, and the fourth-subtype channel-ordered feature map according to the element quantity, to obtain the second reorganized feature map, and specifically to:

[0309] According to the element quantity, the feature elements in each channel feature map included in the third type channel ordering feature map, the third sub-type channel ordering feature map and the fourth sub-type channel ordering feature map are subjected to feature reorganization processing, so as to convert the third type channel ordering feature map, the third sub-type channel ordering feature map and the fourth sub-type channel ordering feature map from two-dimensional tensors into three-dimensional tensors respectively, and third type channel conversion feature maps, third sub-type channel conversion feature maps and fourth sub-type channel conversion feature maps are obtained.

[0310] According to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map, the second reorganized feature map is generated.

[0311] The height parameter of the three-dimensional tensor is the element quantity, and the width parameter of the three-dimensional tensor is the feature element category quantity determined according to the element quantity in each channel feature map and the element quantity.

[0312] In a possible implementation, the multi-type channel feature map includes a third type channel feature map, a fourth type channel feature map and a fifth type channel feature map; the first reorganized feature map includes a third type channel conversion feature map corresponding to the third type channel feature map, and a first sub-type channel conversion feature map and a second sub-type channel conversion feature map corresponding to the fourth type channel feature map; the second reorganized feature map includes a third type channel conversion feature map corresponding to the third type channel feature map, and a third sub-type channel conversion feature map and a fourth sub-type channel conversion feature map corresponding to the fifth type channel feature map; the processor 1801 loads and executes the computer program stored in the memory 1804, and is configured to generate an attention map according to the first reorganized feature map and the second reorganized feature map, and specifically configured to:

[0313] generate a first attention map according to the third type channel conversion feature map, the first sub-type channel conversion feature map and the second sub-type channel conversion feature map, and generate a second attention map according to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map;

[0314] fuse the first attention map and the second attention map to obtain the attention map.

[0315] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to generate a first attention map according to the third type channel conversion feature map, the first sub-type channel conversion feature map, and the second sub-type channel conversion feature map, and generate a second attention map according to the third type channel conversion feature map, the third sub-type channel conversion feature map, and the fourth sub-type channel conversion feature map, specifically for:

[0316] performing first fusion processing on the first sub-type channel conversion feature map and the second sub-type channel conversion feature map to obtain a first fusion feature map, and performing second fusion processing on the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map to obtain a second fusion feature map;

[0317] performing first normalization processing on the first fusion feature map and the third type channel conversion feature map to obtain the first attention map, and performing second normalization processing on the second fusion feature map and the third type channel conversion feature map to obtain the second attention map.

[0318] In a possible implementation, the to-be-processed image is a training image in a training sample, and the training sample further includes a label image corresponding to the to-be-processed image; the processor 1801 loads and executes a computer program stored in the memory 1804, and is further configured to:

[0319] determine first loss data according to a difference between the restored image and the label image;

[0320] determine second loss data according to a correlation between each image block in the restored image and each image block in the label image;

[0321] determine target loss data according to a sum of the first loss data and the second loss data;

[0322] train an initial image processing model based on the target loss data to obtain a target image processing model.

[0323] In a possible implementation, the processor 1801 loads and executes a computer program stored in the memory 1804, to determine second loss data according to a correlation between each image block in the restored image and each image block in the label image, specifically for:

[0324] determine a first pixel value mean and a first pixel value standard deviation of the restored image according to pixel values of each pixel point in the restored image, and determine a second pixel value mean and a second pixel value standard deviation of the label image according to pixel values of each pixel point in the label image;

[0325] According to the first pixel value mean, the first pixel value standard deviation, the second pixel value mean and the second pixel value standard deviation, a Pearson correlation coefficient is calculated, and the second loss data is determined according to the Pearson correlation coefficient.

[0326] In the technical solutions provided in some embodiments of the present application, by extracting image features of the to-be-processed image, a feature map of the degraded image is obtained, and then the feature elements included in the feature map are sorted according to the size relationship of the feature elements, to obtain a sorted feature map, and then the sorted feature map is subjected to first feature reorganization processing according to a preset number of feature element categories, to obtain a first reorganized feature map, and the sorted feature map is subjected to second feature reorganization processing according to a preset number of elements in each feature element category, to obtain a second reorganized feature map; and then an attention map is generated according to the first reorganized feature map and the second reorganized feature map, and the to-be-processed image is reconstructed according to the attention map, to obtain a recovered image corresponding to the to-be-processed image. As can be seen, the pixels affected by the weather are usually close in pixel value, and the sorting processing can determine the pixels that are close in pixel value and dispersed in space, so that the clear pixels and the affected pixels are separated, which is conducive to extracting clear image features and degraded image features. By classifying the sorted pixels based on two different ways respectively, clear features and degraded features of different dimensions can be extracted respectively, an attention map that focuses on preserving clear features and recovering degraded features can be generated, and thus the image recovery effect can be improved to a certain extent.

[0327] It should be understood that, in the embodiments of the present application, the processor 1801 can be a central processing unit (CPU), and the processor 1801 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0328] The present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program includes program instructions, and the program instructions are executed by a processor to perform the steps performed in all embodiments.

[0329] The embodiment of the present application further provides a computer program product or a computer program, the computer program product or the computer program comprising computer instructions stored in a computer readable storage medium, and the computer instructions are executed by a processor of a computer device to perform the method in all the above embodiments.

[0330] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The above-mentioned program can be stored in a computer readable storage medium, and when the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0331] The above disclosure is only a preferred embodiment of the present application, and of course cannot limit the scope of the right of the present application. A person of ordinary skill in the art can understand that all or part of the processes of the above-mentioned embodiment can be implemented, and equivalent changes made according to the claims of the present application still belong to the scope of the present application.

[0332] In addition, it is particularly necessary to point out that when the above embodiments of the present application are applied to specific products or technologies, if it is necessary to obtain the data of a user, the permission or consent of the user needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in relevant countries and regions.

Claims

1. An image processing method, characterized by, The method comprises the steps of: extracting image features of a to-be-processed image to obtain a feature map corresponding to the to-be-processed image; sorting the feature elements included in the feature map according to the size relationship of the feature elements to obtain a sorted feature map; performing first feature reorganization processing on the sorted feature map according to a preset number of feature element categories to obtain a first reorganized feature map, and performing second feature reorganization processing on the sorted feature map according to a preset number of elements under each feature element category to obtain a second reorganized feature map; generating an attention map according to the first reorganized feature map and the second reorganized feature map, and performing reconstruction processing on the to-be-processed image according to the attention map to obtain a restored image corresponding to the to-be-processed image.

2. The method of claim 1, wherein, The method comprises the steps of: performing first extraction processing on the image features of the to-be-processed image to obtain an initial feature map corresponding to the to-be-processed image; sorting the feature elements included in the initial feature map in the horizontal direction and the vertical direction according to the size relationship of the feature elements to obtain an initial sorted feature map; performing second extraction processing on the image features of the initial sorted feature map to obtain the feature map.

3. The method of claim 2, wherein, The initial feature map comprises a plurality of channel feature maps corresponding to a plurality of channels respectively; the method comprises the steps of: performing channel equalization processing on the plurality of channel feature maps included in the initial feature map to obtain a first type of channel feature map and a second type of channel feature map; sorting the feature elements in each channel feature map included in the first type of channel feature map in the horizontal direction and the vertical direction according to the size relationship of the feature elements included in the first type of channel feature map to obtain a first type of channel sorted feature map; performing series processing on the first type of channel sorted feature map and the second type of channel feature map in the corresponding channels to obtain the initial sorted feature map.

4. The method of claim 3, wherein, The attention map comprises a plurality of channel feature maps corresponding to a plurality of channels respectively; the method comprises the steps of: performing channel equalization processing on the plurality of channel feature maps included in the attention map to obtain a first type of channel attention map and a second type of channel attention map; performing restoration processing on the first type of channel attention map and the second type of channel attention map according to a reference sorting index to obtain a first restored attention map and a second restored attention map, the reference sorting index being used to indicate the element position of the feature elements included in the first type of channel sorted feature map in the first type of channel feature map; performing series processing on the first restored attention map and the second restored attention map in the corresponding channels to obtain a target attention map; Reconstruct the to-be-processed image according to the target attention map to obtain a restored image corresponding to the to-be-processed image.

5. The method of claim 4, wherein, The reconstructing the to-be-processed image according to the target attention map to obtain a restored image corresponding to the to-be-processed image comprises: performing third extraction processing on image features of the target attention map to obtain a first attention feature map corresponding to the target attention map, and performing fourth extraction processing on the image features of the target attention map to obtain a second attention feature map corresponding to the target attention map, wherein the feature extraction scales of the third extraction processing and the fourth extraction processing are different; performing activation processing on the first attention feature map according to the first attention feature map to obtain a gating feature map; performing fusion processing on the gating feature map and the second attention feature map to obtain the restored image.

6. The method of claim 5, wherein, The target attention map comprises a plurality of channel feature maps corresponding to a plurality of channels respectively; the third extraction processing on the image features of the target attention map to obtain the first attention feature map and the fourth extraction processing on the image features of the target attention map to obtain the second attention feature map comprise: performing channel equalization processing on a plurality of channel feature maps included in the target attention map according to the plurality of channel feature maps included in the target attention map to obtain a first target class channel attention map and a second target class channel attention map; performing the third extraction processing on image features of the first target class channel attention map to obtain the first attention feature map, and performing the fourth extraction processing on image features of the second target class channel attention map to obtain the second attention feature map.

7. The method of claim 1, wherein, The feature map comprises a plurality of channel feature maps corresponding to a plurality of channels respectively; the sorting processing on feature elements included in the feature map according to a size relationship of the feature elements included in the feature map to obtain a sorted feature map comprises: performing channel equalization processing on a plurality of channel feature maps included in the feature map according to the plurality of channel feature maps included in the feature map to obtain a plurality of class channel feature maps; performing sorting processing on feature elements included in the plurality of class channel feature maps respectively according to a size relationship of the feature elements included in the plurality of class channel feature maps to obtain sorted feature maps corresponding to the plurality of class channel feature maps respectively.

8. The method of claim 7, wherein, The sorting processing on feature elements included in the plurality of class channel feature maps respectively according to a size relationship of the feature elements included in the plurality of class channel feature maps to obtain sorted feature maps corresponding to the plurality of class channel feature maps respectively comprises: performing feature reorganization processing on the feature elements in the plurality of channel feature maps included in the plurality of class channel feature maps respectively to convert the plurality of class channel feature maps from three-dimensional tensors to two-dimensional tensors respectively to obtain channel conversion feature maps corresponding to the plurality of class channel feature maps respectively; According to the size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorted feature map corresponding to each of the multi-class channel feature maps.

9. The method of claim 8, wherein, According to the size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorted feature map corresponding to each of the multi-class channel feature maps. According to the size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorted feature map corresponding to each of the multi-class channel feature maps. According to the size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorted feature map corresponding to each of the multi-class channel feature maps.

10. The method of claim 7, wherein, According to the size relationship of each feature element included in the channel conversion feature map corresponding to each of the multi-class channel feature maps, the channel conversion feature map corresponding to each of the multi-class channel feature maps is respectively sorted to obtain a channel sorted feature map corresponding to each of the multi-class channel feature maps. The multi-class channel feature maps include a third-class channel feature map, a fourth-class channel feature map, and a fifth-class channel feature map; the first feature reorganization processing of the sorted feature map according to the preset feature element category number and the second feature reorganization processing of the sorted feature map according to the element number under each feature element category to obtain a first reorganization feature map and a second reorganization feature map, include: According to the fourth-class channel sorted feature map including a plurality of channel feature maps, the fourth-class channel sorted feature map including a plurality of channel feature maps is processed by channel equalization to obtain a first sub-class channel sorted feature map and a second sub-class channel sorted feature map. According to the fifth-class channel sorted feature map including a plurality of channel feature maps, the fifth-class channel sorted feature map including a plurality of channel feature maps is processed by channel equalization to obtain a third sub-class channel sorted feature map and a fourth sub-class channel sorted feature map. According to the third-class channel sorted feature map, the first sub-class channel sorted feature map, and the second sub-class channel sorted feature map, the first feature reorganization processing is performed according to the feature element category number to obtain the first reorganization feature map. According to the third-class channel sorted feature map, the third sub-class channel sorted feature map, and the fourth sub-class channel sorted feature map, the second feature reorganization processing is performed according to the element number to obtain the second reorganization feature map.

11. The method of claim 10, wherein, The element number under each feature element category corresponding to the first reorganization feature map is greater than the element number under each feature element category corresponding to the second reorganization feature map. According to the third-class channel sorted feature map, the first sub-class channel sorted feature map, and the second sub-class channel sorted feature map, the first feature reorganization processing is performed according to the feature element category number to obtain the first reorganization feature map. According to the feature element category quantity, feature elements in each channel feature map included in the third type channel sorting feature map, the first sub-type channel sorting feature map and the second sub-type channel sorting feature map are subjected to feature reorganization processing, so as to convert the third type channel sorting feature map, the first sub-type channel sorting feature map and the second sub-type channel sorting feature map from two-dimensional tensors into three-dimensional tensors respectively, and third type channel conversion feature maps, first sub-type channel conversion feature maps and second sub-type channel conversion feature maps are obtained; According to the third type channel conversion feature map, the first sub-type channel conversion feature map and the second sub-type channel conversion feature map, the first reorganization feature map is generated; The height parameter of the three-dimensional tensor is the feature element category quantity, and the width parameter of the three-dimensional tensor is the element quantity in each feature element category determined according to the element quantity in the channel feature map and the feature element category quantity.

12. The method of claim 10, wherein, The second feature reorganization processing of the third type channel sorting feature map, the third sub-type channel sorting feature map and the fourth sub-type channel sorting feature map according to the element quantity to obtain the second reorganization feature map, comprising: According to the element quantity, feature elements in each channel feature map included in the third type channel sorting feature map, the third sub-type channel sorting feature map and the fourth sub-type channel sorting feature map are subjected to feature reorganization processing, so as to convert the third type channel sorting feature map, the third sub-type channel sorting feature map and the fourth sub-type channel sorting feature map from two-dimensional tensors into three-dimensional tensors respectively, and third type channel conversion feature maps, third sub-type channel conversion feature maps and fourth sub-type channel conversion feature maps are obtained; According to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map, the second reorganization feature map is generated; The height parameter of the three-dimensional tensor is the element quantity, and the width parameter of the three-dimensional tensor is the feature element category quantity determined according to the element quantity in the channel feature map and the element quantity.

13. The method of claim 7, wherein, The multi-type channel feature map includes a third type channel feature map, a fourth type channel feature map and a fifth type channel feature map; the first reorganization feature map includes a third type channel conversion feature map corresponding to the third type channel feature map, and a first sub-type channel conversion feature map and a second sub-type channel conversion feature map corresponding to the fourth type channel feature map; the second reorganization feature map includes a third type channel conversion feature map corresponding to the third type channel feature map, and a third sub-type channel conversion feature map and a fourth type sub-channel conversion feature map corresponding to the fifth type channel feature map; The attention map is generated according to the first reorganization feature map and the second reorganization feature map, comprising: generate a first attention map according to the third type channel conversion feature map, the first sub-type channel conversion feature map and the second sub-type channel conversion feature map, and generate a second attention map according to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map; perform fusion processing on the first attention map and the second attention map to obtain the attention map.

14. The method of claim 13, wherein, The method according to the third type channel conversion feature map, the first sub-type channel conversion feature map and the second sub-type channel conversion feature map, and generate a first attention map according to the third type channel conversion feature map, the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map, includes: perform first fusion processing on the first sub-type channel conversion feature map and the second sub-type channel conversion feature map to obtain a first fusion feature map, and perform second fusion processing on the third sub-type channel conversion feature map and the fourth sub-type channel conversion feature map to obtain a second fusion feature map; perform first normalization processing on the first fusion feature map and the third type channel conversion feature map to obtain the first attention map, and perform second normalization processing on the second fusion feature map and the third type channel conversion feature map to obtain the second attention map.

15. The method according to any one of claims 1 to 14, characterized in that, The to-be-processed image is a training image in a training sample, and the training sample further includes a label image corresponding to the to-be-processed image; the method further includes: determine first loss data according to a difference between the restored image and the label image; determine second loss data according to a correlation between each image block in the restored image and each image block in the label image; determine target loss data according to a sum of the first loss data and the second loss data; train an initial image processing model based on the target loss data to obtain a target image processing model.

16. The method of claim 15, wherein, The method according to a correlation between each image block in the restored image and each image block in the label image to determine second loss data includes: determine a first pixel value mean and a first pixel value standard deviation of the restored image according to pixel values of each pixel point in the restored image, and determine a second pixel value mean and a second pixel value standard deviation of the label image according to pixel values of each pixel point in the label image; calculate a Pearson correlation coefficient according to the first pixel value mean, the first pixel value standard deviation, the second pixel value mean and the second pixel value standard deviation, and determine the second loss data according to the Pearson correlation coefficient.

17. An image processing apparatus characterized by comprising: The method includes: extract an image feature of a to-be-processed image to obtain a feature map corresponding to the to-be-processed image; perform sorting processing on feature elements included in the feature map according to a size relationship of each feature element included in the feature map to obtain a sorted feature map; The processing unit is configured to perform first feature reorganization processing on the sorted feature map according to a preset number of feature element categories, to obtain a first reorganized feature map, and perform second feature reorganization processing on the sorted feature map according to a preset number of elements in each feature element category, to obtain a second reorganized feature map. The generating unit is configured to generate an attention map according to the first reorganized feature map and the second reorganized feature map, and perform reconstruction processing on the to-be-processed image according to the attention map, to obtain a restored image corresponding to the to-be-processed image.

18. An image processing apparatus characterized by comprising: The image processing device comprises: one or more processors; a memory configured to store one or more computer programs, which, when executed by the one or more processors, cause the image processing device to implement the image processing method of any one of claims 1-16.

19. A computer readable medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the image processing method of any one of claims 1-16.

20. A computer program product, characterised in that, The computer program product comprises a computer program stored in a computer readable storage medium, and the processor of the electronic device reads and executes the computer program from the computer readable storage medium, so that the electronic device executes the image processing method of any one of claims 1-16.

Images