Image processing method and device, model training method and device, equipment and storage medium

Abstract

The invention provides an image processing method, relates to the technical field of artificial intelligence, in particular to the technical field of chips and the technical field of computer vision, and can be applied to image defogging and image high-resolution reconstruction scenes. The specific implementation scheme is as follows: according to image block features of a to-be-processed image, determining to-be-processed features corresponding to the image block features; multiple scanning sequence features are obtained according to multiple scanning modes and the to-be-processed features, the multiple scanning modes comprise a first scanning mode, the first scanning mode is used for indicating scanning in the scanning direction, and the scanning direction is the direction forming a scanning angle with a target dimension in multiple dimensions of the to-be-processed features; processing the plurality of scanning sequence features based on an attention mechanism to obtain processed features corresponding to the image block features; and decoding the coding features to obtain a processed image, the coding features being obtained according to the processed features. The invention further provides a model training method and device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of chip technology and computer vision technology, and can be applied to image defogging and image high-resolution reconstruction scenarios. More specifically, the present disclosure provides an image processing method, an image processing model training method, a device, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the quality of images can be improved by artificial intelligence technology to improve the resolution or clarity of images. Summary of the invention

[0003] The present disclosure provides an image processing method, a training method, an apparatus, a device and a storage medium for an image processing model.

[0004] According to one aspect of the present disclosure, an image processing method is provided, the method comprising: determining features to be processed corresponding to the image block features according to image block features of an image to be processed; obtaining multiple scanning sequence features according to multiple scanning modes and features to be processed, wherein the features to be processed are features of multiple dimensions, the multiple scanning modes include a first scanning mode, the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction forming a scanning angle with a target dimension in the multiple dimensions of the features to be processed; processing the multiple scanning sequence features based on an attention mechanism to obtain processed features corresponding to the image block features; decoding the encoded features to obtain a processed image, wherein the encoded features are obtained based on the processed features.

[0005] According to another aspect of the present disclosure, a training method for an image processing model is provided, wherein the image processing model includes a coding sub-model and a decoding sub-model, wherein the coding sub-model includes a coding network, wherein the coding network includes a first processing module, a scanning module and an attention processing module, and the method includes: inputting image block features of a sample image to be processed into the first processing module to obtain features to be processed corresponding to the image block features; inputting the features to be processed into the scanning module to obtain a plurality of scanning sequence features, wherein the scanning module corresponds to a plurality of scanning modes, and the features to be processed are features of a plurality of dimensions, wherein the plurality of scanning modes include a first scanning mode, and the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction forming a scanning angle with a target dimension in a plurality of dimensions of the features to be processed; inputting the plurality of scanning sequence features into the attention processing module to obtain processed features corresponding to the image block features; inputting the coding features into the decoding sub-model to obtain a processed sample image, wherein the coding features are obtained based on the processed features; and training the image processing model based on the sample image and the processed sample image.

[0006] According to another aspect of the present disclosure, an image processing device is provided, the device comprising: a determination module, for determining, based on image block features of an image to be processed, features to be processed corresponding to image block features; a first acquisition module, for obtaining, based on a plurality of scanning modes and features to be processed, a plurality of scanning sequence features, wherein the features to be processed are features of a plurality of dimensions, the plurality of scanning modes comprising a first scanning mode, the first scanning mode being used to indicate scanning along a scanning direction, the scanning direction being a direction forming a scanning angle with a target dimension in the plurality of dimensions of the features to be processed; a second acquisition module, for processing the plurality of scanning sequence features based on an attention mechanism, to obtain processed features corresponding to the image block features; a decoding module, for decoding the encoded features, to obtain a processed image, wherein the encoded features are obtained based on the processed features.

[0007] According to another aspect of the present disclosure, a training device for an image processing model is provided, the image processing model includes an encoding sub-model and a decoding sub-model, the encoding sub-model includes an encoding network, the encoding network includes a first processing module, a scanning module and an attention processing module, the device includes: a third acquisition module, used to input the image block features of the sample image to be processed into the first processing module, and obtain the features to be processed corresponding to the image block features; a fourth acquisition module, used to input the features to be processed into the scanning module, and obtain a plurality of scanning sequence features, wherein the scanning module corresponds to a plurality of scanning modes, the features to be processed are features of a plurality of dimensions, the plurality of scanning modes include a first scanning mode, the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction forming a scanning angle with a target dimension in the plurality of dimensions of the features to be processed; a fifth acquisition module, used to input the plurality of scanning sequence features into the attention processing module, and obtain processed features corresponding to the image block features; a sixth acquisition module, used to input the encoding features into the decoding sub-model, and obtain a processed sample image, wherein the encoding features are obtained based on the processed features; a training module, used to train the image processing model based on the sample image and the processed sample image.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and apparatus can be applied according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0015] Figure 3 is an exemplary schematic diagram of an image processing method according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of processing a feature to be processed according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of obtaining visual state features according to an embodiment of the present disclosure;

[0018] Fig. 6A is a schematic diagram of a feature to be scanned according to an embodiment of the present disclosure;

[0019] Figure 6B is a schematic diagram of obtaining attention features according to an embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram of an adaptive attention mechanism according to an embodiment of the present disclosure;

[0021] Figure 8 is an exemplary schematic diagram of an image processing method according to another embodiment of the present disclosure;

[0022] Fig. 9 is a flowchart of a method for training an image processing model according to another embodiment of the present disclosure;

[0023] Fig.10 is a schematic diagram of an image processing model according to an embodiment of the present disclosure;

[0024] Fig.11is a schematic diagram of a coding network according to an embodiment of the present disclosure;

[0025] Fig.12 is a schematic diagram of an image processing model according to another embodiment of the present disclosure;

[0026] Fig.13 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

[0027] Fig.14 is a block diagram of a training apparatus for an image processing model according to another embodiment of the present disclosure; and

[0028] Fig.15 is a block diagram of an electronic device to which at least one of an image processing method and a training method of an image processing model can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] In some embodiments, low-quality, unclear or low-resolution images may include foggy images. Foggy images can be defogged to obtain images with higher clarity. For example, image defogging can be performed based on a convolutional neural network (CNN). However, the global modeling capability of convolutional neural networks is insufficient, and it is difficult to accurately express the features of the entire image, and the defogging effect is difficult to fully improve. For another example, the Transformer network can obtain long-distance dependencies between multiple features in an image and can be used for image defogging. However, the computational complexity of the Transformer network is high, and the computing resource overhead is large. Both convolutional neural networks and Transformer networks find it difficult to achieve an ideal balance between global scene reconstruction, local detail retention, and computational efficiency.

[0031] Therefore, in order to efficiently improve image quality, clarity and resolution, the present disclosure provides an image processing method, which will be described below.

[0032] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] It should be noted that the image processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The image processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0037] It can be understood that the above describes the system architecture of the present disclosure, and the following describes the method of the present disclosure.

[0038] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the method 200 may include operations S210 to S240.

[0040] In operation S210, based on the image block feature of the image to be processed, a feature to be processed corresponding to the image block feature is determined.

[0041] In the disclosed embodiment, various processes may be performed on the image patch features to determine the features to be processed. For example, the image patch features may be used as the features to be processed.

[0042] In operation S220, a plurality of scanning sequence features are obtained according to the plurality of scanning modes and features to be processed.

[0043] In the embodiment of the present disclosure, the feature to be processed may be a feature of multiple dimensions. For example, the feature to be processed may be a feature of K dimensions. K may be an integer greater than 1.

[0044] In an embodiment of the present disclosure, the plurality of scanning modes include a first scanning mode. The first scanning mode may indicate scanning along a scanning direction. The scanning direction may be a direction that forms a scanning angle with a target dimension among multiple dimensions of the feature to be processed. For example, by scanning features of K dimensions, scanning sequence features of M dimensions may be obtained. M may be an integer less than K. Taking K=2 as an example, M may be 1. The scanning angle may be any angle value, and may be an acute angle, an obtuse angle, a right angle, etc.

[0045] In operation S230, a plurality of scanning sequence features are processed based on an attention mechanism to obtain processed features corresponding to the image block features.

[0046] In the embodiment of the present disclosure, based on the attention mechanism, multiple attention weights of multiple scanning sequence features can be determined. The multiple scanning sequence features are weighted by using the multiple attention weights to obtain processed features. The weighted processing can be weighted summation, weighted average, etc.

[0047] In operation S240, the encoded features are decoded to obtain a processed image.

[0048] In the disclosed embodiment, the coding feature may be obtained based on the processed feature. For example, the processed feature may be used as the coding feature.

[0049] In the embodiments of the present disclosure, decoding may be performed based on various decoding methods, such as deconvolution, upsampling, etc.

[0050] In the embodiment of the present disclosure, compared with the image to be processed, the processed image may be an image of higher quality, higher definition or higher resolution.

[0051] Through the embodiments of the present disclosure, scanning is performed based on multiple scanning modes, and the connection relationship between the global spatial features of the image in different directions can be fully obtained. The attention mechanism is used to process multiple scanning sequence features obtained based on multiple scanning modes, which can fully utilize the features in different directions of the image, and can fully utilize the characteristics of the image to be processed itself to process various styles and types of images. Based on multiple scanning modes and attention mechanisms, key features can be extracted, and decoding can be performed based on key processed features, so that high-quality, higher-definition or higher-resolution images can be efficiently generated.

[0052] It can be understood that the method of the present disclosure is described above, and the method of the present disclosure will be further described below.

[0053] Figure 3 is an exemplary schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0054] like Figure 3 As shown, the image to be processed image30 may be a foggy image. The scale of the image to be processed image30 may be H×W×C. H represents the height of the image. W represents the width of the image. C represents the number of channels of the image. The image to be processed image30 is encoded en30 to obtain encoding features. The encoding en30 may include the above-mentioned operations S210 to S230. The encoded features are decoded de30 to obtain a processed image image31. The processed image image31 may be a defogged image. The decoding de30 may include the above-mentioned operation S240.

[0055] It can be understood that the above describes the principle of the present disclosure, and the method of the present disclosure will be further described below.

[0056] In some embodiments, the above method may further include: upsampling the image to be processed to obtain an upsampled image. For example, the image to be processed may be pixel shuffled or interpolated to obtain an upsampled image. Through the embodiments of the present disclosure, the image resolution may be effectively improved. In addition, compared with interpolation processing, pixel shuffle may better preserve the detail information of the image and may also reduce information loss.

[0057] Next, in some embodiments, the upsampled image may be subjected to a patch embedding process to obtain image block features of the image to be processed. For example, the patch embedding process includes: determining multiple image blocks of the upsampled image. The size of the image block may be smaller than the size of the upsampled image. The multiple image blocks are linearly projected into a high-dimensional feature space to obtain image block features of the image to be processed. Thus, the image information can be efficiently processed, and the local structural features of the image can be captured, thereby improving the effect of subsequent processing. Next, the image block features may be processed to obtain processed features, which will be described below.

[0058] Figure 4 It is a schematic diagram of processing a feature to be processed according to an embodiment of the present disclosure.

[0059] In some embodiments, in some implementations of the above operation S210, determining the feature to be processed corresponding to the image block feature according to the image block feature of the image to be processed includes: using at least one convolution kernel to convolve the image block feature to obtain at least one convolution result. According to the at least one convolution result, obtaining the feature to be processed corresponding to the image block feature. Figure 4 As shown, before processing the image block feature based on at least one convolution kernel, the image block feature pf40 can be preprocessed conv40 based on the preprocessing convolution kernel to obtain a preprocessing result. Based on the first convolution kernel, the preprocessing result is subjected to a first convolution processing conv41 to obtain a first convolution result. The first convolution result is subjected to batch normalization processing (batchnorm) bn41 to obtain a feature to be processed. The preprocessing convolution kernel can be a 1×1 convolution kernel. The first convolution kernel can be a 5×5 convolution kernel. It can be understood that Figure 4 The present disclosure is explained by taking at least one convolution kernel including a first convolution kernel as an example. However, the present disclosure is not limited to this, and the at least one convolution kernel may also be multiple. The convolution results obtained by processing the preprocessing results using different convolution kernels can be fused to obtain the features to be processed. Through the embodiments of the present disclosure, based on at least one convolution kernel, the features of an image of at least one scale can be obtained, and the features of the image content at different scales can be fully utilized. In this way, the local spatial information of the image can be better preserved, which helps to improve the image processing effect. Normalizing the convolution results can effectively improve the image processing efficiency and help to achieve the convergence of the processing results.

[0060] In some embodiments, based on the features to be processed, the processed features can be obtained. Figure 4As shown, by performing the above operations S220 and S230 on the feature to be processed, a visual state feature vss40 can be obtained. The visual state feature vss40 can be a visual state space (Visual State Space, VSS) feature. According to the visual state feature vss40, a processed feature can be obtained, which will be described below.

[0061] In the disclosed embodiment, the visual state feature is normalized to obtain the normalized visual state feature. The normalized visual state is subjected to multi-layer perception processing to obtain the multi-layer perception feature. The multi-layer perception feature is fused with the image block feature to obtain the processed feature. Figure 4 As shown, the normalization process may include a layer normalization (layernorm) process ln40. The visual state feature vss40 is subjected to a layer normalization process ln40 to obtain a normalized visual state feature. The multi-layer perception process mlp40 may be implemented based on a multi-layer perceptron (MLP). The normalized visual state feature is subjected to a multi-layer perception process mlp40 to obtain a multi-layer perception feature. The multiple perception features are fused with the image block feature pf40 to obtain a processed feature f41. The fusion method may include addition.

[0062] It can be understood that the encoding method of the present disclosure is described above, and the method of obtaining visual state features during the encoding process will be further described below.

[0063] Figure 5 is a schematic diagram of obtaining visual state features according to an embodiment of the present disclosure.

[0064] In some embodiments, in some implementations of the above operation S220, obtaining a plurality of scanning sequence features according to a plurality of scanning modes and features to be processed may include: processing the features to be processed to obtain the features to be scanned.

[0065] In the embodiment of the present disclosure, linear processing is performed on the feature to be processed to obtain a first linear feature. Depth convolution is performed on the first linear feature to obtain a feature to be scanned. Figure 5As shown, the feature to be processed can be subjected to layer normalization processing ln51 to obtain the normalized feature to be processed. The normalized feature to be processed is subjected to linear processing (linear) l50 to obtain the first linear feature. The first linear feature is subjected to depthwise convolution (depthwise conv) depc50 to obtain the feature to be scanned. The feature to be scanned is a feature of multiple dimensions, and the multiple dimensions of the feature to be scanned correspond to the multiple dimensions of the processing feature. The dimension of the feature to be scanned can also be K. Through the embodiments of the present disclosure, linear processing and depthwise convolution are performed on the feature to be processed, which helps to improve the scanning efficiency, thereby improving the image processing efficiency.

[0066] In some embodiments, in some implementations of the above operation S220, obtaining multiple scanning sequence features according to multiple scanning modes and features to be processed may also include: scanning the features to be scanned according to multiple scanning modes to obtain multiple scanning sequence features.

[0067] Next, the above operation S230 may be performed. In some embodiments, in some implementations of the above operation S230, processing multiple scan sequence features based on the attention mechanism to obtain processed features corresponding to the image block features may include: processing multiple scan sequence features based on the attention mechanism to obtain attention features. Figure 5 As shown, after the first linear feature is subjected to a deep convolution process depc50, a feature to be scanned can be obtained. The feature to be scanned can be subjected to an adaptive selective scan ass50. The adaptive selective scan ass50 may include: scanning the feature to be scanned according to a plurality of scanning modes to obtain a plurality of scanning sequence features. And, processing the plurality of scanning sequence features based on an attention mechanism to obtain an attention feature. The adaptive selective scan ass50 may be an adaptive two-dimensional selective scan (Adaptive 2D-Selective-Scan, A-SS2D).

[0068] In some embodiments, in some implementations of the above operation S230, processing multiple scan sequence features based on the attention mechanism to obtain processed features corresponding to the image block features may also include: obtaining the processed features based on the attention features.

[0069] In the embodiment of the present disclosure, obtaining the processed feature according to the attention feature may include: normalizing the attention feature to obtain a normalized attention feature; performing linear processing on the feature to be processed to obtain a second linear feature; fusing the second linear feature with the normalized attention feature to obtain a visual state feature. Figure 5As shown, the normalization process may include layer normalization process ln52. By performing layer normalization process ln52 on the attention feature, a normalized attention feature can be obtained. By performing linear processing l51 on the normalized feature to be processed, a second linear feature can be obtained. By fusing the second linear feature and the normalized attention feature, an attention fusion feature can be obtained. The fusion method may include multiplication. By performing linear processing l52 on the attention fusion feature, a visual state feature can be obtained. The visual state feature may be the above-mentioned visual state feature vss40. Through the embodiment of the present disclosure, the attention feature is normalized, and the feature to be processed is linearly processed, and then the processed results are fused, so that the information in the feature to be processed can be fully utilized, and the information in the attention feature can also be fully utilized to more efficiently realize image processing such as image defogging and high-resolution reconstruction.

[0070] It can be understood that the above describes the manner in which the present disclosure obtains visual state features, and some manners of various scanning modes will be described below.

[0071] In some embodiments, the plurality of scanning modes may include a first scanning mode and a second scanning mode. The first scanning mode may indicate scanning along a scanning direction. The second scanning mode may indicate scanning based on a sliding window. Fig. 6A and Figure 6B Provide further explanation.

[0072] Fig. 6A is a schematic diagram of features to be scanned according to an embodiment of the present disclosure.

[0073] like Fig. 6A As shown, the feature to be scanned nsf60 can be represented as a 3×3 feature map. The feature to be scanned may include multiple dimensions and may include a first dimension and a second dimension. The first dimension corresponds to the height direction of the feature to be scanned nsf60. The second dimension corresponds to the width direction of the feature to be scanned nsf60. It can be understood that the scale (shape) of the feature to be scanned nsf60 can be I×J, where I is an integer greater than 1 and J is an integer greater than 1. Fig. 6A The I=3 and J=3 shown are only examples. It can also be understood that the feature to be scanned can be a feature of three dimensions or higher dimensions. Fig. 6A The two-dimensional features to be scanned are shown as examples only.

[0074] like Fig. 6AAs shown, the feature to be scanned nsf60 may include sub-feature to be scanned nsf601, sub-feature to be scanned nsf602, sub-feature to be scanned nsf603, sub-feature to be scanned nsf604, sub-feature to be scanned nsf605, sub-feature to be scanned nsf606, sub-feature to be scanned nsf607, sub-feature to be scanned nsf608, and sub-feature to be scanned nsf609.

[0075] In the embodiment of the present disclosure, there may be multiple first scanning modes. Multiple first scanning modes correspond to multiple different scanning angles. Fig. 6A As shown, the target dimension is the first dimension as an example. As mentioned above, the first dimension corresponds to the height direction. The height direction can be the direction from the sub-feature to be scanned nsf607 to the sub-feature to be scanned nsf601. The scanning angle can be the angle formed by the scanning direction and the height direction.

[0076] In the disclosed embodiment, the feature to be scanned is scanned according to the first scanning mode to obtain the first scanning sequence feature. For example, the feature to be scanned is scanned according to multiple first scanning modes respectively to obtain multiple first scanning sequence features.

[0077] In the disclosed embodiment, the scanning angle may be greater than 0 degrees and less than 90 degrees. For example, the scanning angle may be 45 degrees, that is, the corresponding first scanning mode may indicate scanning from the lower right to the upper left of the feature to be scanned nsf60. Thus, scanning according to the first scanning mode may sequentially include sub-feature to be scanned nsf609, sub-feature to be scanned nsf605, sub-feature to be scanned nsf601, sub-feature to be scanned nsf606, sub-feature to be scanned nsf602, sub-feature to be scanned nsf603, sub-feature to be scanned nsf608, sub-feature to be scanned nsf604, and sub-feature to be scanned nsf607.

[0078] In the disclosed embodiment, the scanning angle may be greater than 90 degrees and less than 180 degrees. For example, the scanning angle may be 135 degrees, and the corresponding first scanning mode may indicate scanning from the upper right to the lower left of the feature to be scanned nsf60. Thus, scanning according to the first scanning mode may sequentially include sub-feature to be scanned nsf603, sub-feature to be scanned nsf605, sub-feature to be scanned nsf607, sub-feature to be scanned nsf602, sub-feature to be scanned nsf604, sub-feature to be scanned nsf601, sub-feature to be scanned nsf606, sub-feature to be scanned nsf608, and sub-feature to be scanned nsf609.

[0079] In the disclosed embodiment, the scanning angle may be greater than 180 degrees and less than 270 degrees. For example, the scanning angle may be 225 degrees, and the corresponding first scanning mode may indicate scanning from the upper left to the lower right of the feature to be scanned nsf60. Thus, scanning according to the first scanning mode may sequentially include sub-feature to be scanned nsf601, sub-feature to be scanned nsf605, sub-feature to be scanned nsf609, sub-feature to be scanned nsf602, sub-feature to be scanned nsf606, sub-feature to be scanned nsf603, sub-feature to be scanned nsf604, sub-feature to be scanned nsf608, and sub-feature to be scanned nsf607.

[0080] In the disclosed embodiment, the scanning angle may be greater than 270 degrees and less than 360 degrees. For example, the scanning angle may be 315 degrees, and the corresponding first scanning mode may indicate scanning from the lower left to the upper right of the feature to be scanned nsf60. Thus, scanning according to the first scanning mode may obtain the first scanning sequence feature may include sub-feature to be scanned nsf607, sub-feature to be scanned nsf605, sub-feature to be scanned nsf603, sub-feature to be scanned nsf604, sub-feature to be scanned nsf602, sub-feature to be scanned nsf601, sub-feature to be scanned nsf608, sub-feature to be scanned nsf606, and sub-feature to be scanned nsf609 in sequence. It can be understood that in the case where the feature to be scanned includes more sub-features to be scanned, the scanning angle may also be other values.

[0081] In the disclosed embodiment, the scanning angle can also be 0 degrees, 90 degrees, 270 degrees and 360 degrees, and the corresponding multiple first scanning modes can respectively indicate scanning from bottom to top, scanning from right to left, scanning from top to bottom and scanning from left to right of the nsf60 of the feature to be scanned. Through the disclosed embodiment, scanning can be performed along the scanning direction corresponding to any scanning angle, which greatly enriches the directional scanning method of the image feature and can more comprehensively obtain the global information of the image.

[0082] It can be understood that the first scanning mode of the present disclosure is described above, and the second scanning mode of the present disclosure will be described below.

[0083] In the disclosed embodiment, there may be multiple second scanning modes. The window parameters of the sliding windows of the multiple second scanning modes are different from each other. The window parameters include at least one of the window size, the sliding direction and the sliding step length. The window size may be less than or equal to the scale of the feature to be scanned. The sliding direction may be any direction. The sliding step length may be the length of one or more sub-features to be scanned.

[0084] like Fig. 6AAs shown, the window size may be at least one of 2×2 and 3×3. Taking sliding from left to right with a sliding step of 1 width of a sub-feature to be scanned as an example, the feature to be scanned nsf60 is scanned based on a 2×2 sliding window, and the obtained second scanning sequence features may sequentially include sub-feature to be scanned nsf601, sub-feature to be scanned nsf602, sub-feature to be scanned nsf605, sub-feature to be scanned nsf604, sub-feature to be scanned nsf602, sub-feature to be scanned nsf603, sub-feature to be scanned nsf606, sub-feature to be scanned nsf605, sub-feature to be scanned nsf604, sub-feature to be scanned nsf605, sub-feature to be scanned nsf607, sub-feature to be scanned nsf608, sub-feature to be scanned nsf605, sub-feature to be scanned nsf606, sub-feature to be scanned nsf609, and sub-feature to be scanned nsf608. Through the embodiments of the present disclosure, scanning can be performed based on a sliding window, which can further increase the richness of scanning methods and further comprehensively acquire global information of image features.

[0085] It can be understood that the above describes the method of obtaining the features of the scanning sequence, and the following describes the method of processing based on the attention mechanism.

[0086] Figure 6B is a schematic diagram of obtaining attention features according to an embodiment of the present disclosure.

[0087] like Figure 6B As shown, the features to be scanned can be scanned along different scanning directions. The scanning directions include from left to right, from top to bottom, from top left to bottom right, etc. The features to be scanned can also be scanned based on a 2×2 sliding window and a 3×3 sliding window. Thus, multiple scanning sequence features can be scanned.

[0088] In the embodiment of the present disclosure, processing multiple scan sequence features based on the attention mechanism to obtain processed features corresponding to the image block features may include: fusing the multiple scan sequence features to obtain scan fusion features. Figure 6B As shown, multiple scanning sequence features can be fused by fuse60 to obtain scanning fusion features.

[0089] In an embodiment of the present disclosure, processing multiple scan sequence features based on an attention mechanism to obtain processed features corresponding to image block features may include: performing multiple full-connection processes on scan fusion features to obtain attention weight features. The dimension of the attention weight feature is consistent with the number of scan sequence features. The scan fusion features are weighted using the attention weight features to obtain the attention features. For example, the attention mechanism may be an adaptive attention mechanism (adaptiveattention). The adaptive attention mechanism AATT60 may include performing multiple full-connection processes and weighted processes on the scan fusion features. Thus, based on the adaptive attention mechanism AATT60, the attention feature AATT60 may be obtained. Through the embodiment of the present disclosure, the scan sequence features of multiple different scan directions and sliding windows are fused, and the multiple scan sequence features are weighted based on the attention weights, which can better enhance the association between the global spatial features of the image in different directions.

[0090] It can be understood that the above describes the manner in which the present invention obtains attention features, and the adaptive attention mechanism of the present invention will be further described below.

[0091] Figure 7 is a schematic diagram of an adaptive attention mechanism according to an embodiment of the present disclosure.

[0092] In the disclosed embodiment, performing multiple full-connection processing on the scanned fusion feature to obtain the attention weight feature may include: performing pooling processing on the scanned fusion feature to obtain the pooled feature. Performing a first full-connection processing on the pooled feature to obtain the first fully-connected feature. Performing a first activation processing on the first fully-connected feature to obtain the first activated feature. Performing a second full-connection processing on the first activated feature to obtain the second fully-connected feature. Performing a second activation processing on the second fully-connected feature to obtain the attention weight feature.

[0093] like Figure 7 As shown, the scale of the scanning fusion feature sff70 can be H'×W'×C'. H' represents the height of the scanning fusion feature. W' represents the width of the scanning fusion feature. C' represents the number of channels of the scanning fusion feature. The scale of the scanning fusion feature can be different from or the same as the scale of the image to be processed.

[0094] like Figure 7As shown, the scan fusion feature sff70 can be subjected to a pooling process p70 to obtain a pooling feature. The pooling process p70 can be, for example, average pooling. The scale of the pooling can be 1×1×C'. The pooling feature can be subjected to a first full connection process fc70 to obtain a first full connection feature. The scale of the first full connection feature can be 1×1×(C' / r). r can be an integer greater than or equal to 1. r is the rank corresponding to the first full connection process fc70. The first full connection feature can be subjected to a first activation process relu70 to obtain a first activation feature. The first activation process relu70 can be implemented based on a Rectified Linear Unit (ReLU). The scale of the first activation feature can be 1×1×(C' / r). The first activation feature can be subjected to a second full connection process fc71 to obtain a second full connection feature. The scale of the second full connection feature can be 1×1×C'. The second full connection feature can be subjected to a second activation process sig70 to obtain an attention weight feature. The second activation process sig70 may be implemented based on a sigmoid function. The scale of the attention weight feature may be 1×1×C'.

[0095] Next, the scan fusion feature sff70 can be weighted using the attention weight feature to obtain the attention feature att70. The scale of the attention feature att70 can be H'×W'×C'. It can be understood that based on the attention feature, the processed feature can be obtained. Next, the processed feature can be decoded to obtain the processed image. Through the embodiments of the present disclosure, based on average pooling, full connection, activation and other processing, the importance evaluation of different scan sequence features is realized, the combination of global feature representation and each scan sequence feature is realized, the ability to obtain image features is improved, the feature extraction is more refined, it is helpful to reconstruct high-quality images, and the adaptability to different input images is enhanced.

[0096] It can be understood that the above takes the example of performing level 1 encoding and decoding on the processed image to illustrate the encoding and decoding methods of the present disclosure, but the present disclosure is not limited thereto, and the processed image can be encoded and decoded at N levels, which will be explained below.

[0097] Figure 8 is an exemplary schematic diagram of an image processing method according to another embodiment of the present disclosure.

[0098] In the embodiment of the present disclosure, the above method may further include: upsampling the image to be processed to obtain an upsampled image. Performing block embedding processing on the upsampled image to obtain the first level image block features of the image to be processed. Figure 8As shown, the image to be processed image80 is upsampled u80 to obtain the upsampled image. The scale of the image to be processed image30 can be H×W×C. H represents the height of the image to be processed. W represents the width of the image to be processed. C represents the number of channels of the image to be processed. The upsampled image is subjected to block embedding processing pe80 to obtain the first-level image block features of the image to be processed. Next, the first-level image block features can be encoded en80 to obtain the first-level processed features. The scale of the first-level image block features can be (H / 4)×(W / 4)×2C. It can be understood that the method of obtaining the first-level processed features according to the first-level image block features is the same or similar to the method of obtaining the processed features f41 according to the image block features pf40, and the present disclosure will not repeat them here.

[0099] In the disclosed embodiment, the nth level processed features corresponding to the nth level image block features are subjected to block fusion processing to obtain the n+1th level image block features corresponding to the image to be processed. n is an integer greater than or equal to 1 and less than N, and N is an integer greater than 1.

[0100] like Figure 8 As shown, the first-level processed features can be subjected to a patch merging process pm81 to obtain second-level image block features. The scale of the second-level image block features can be (H / 8)×(W / 8)×4C. Next, the second-level image block features can be subjected to an encoding process en81 to obtain second-level processed features. It can be understood that the manner of obtaining the second-level processed features based on the second-level image block features is the same or similar to the manner of obtaining the processed features f41 based on the image block features pf40, and the present disclosure will not be repeated here.

[0101] like Figure 8 As shown, the second-level processed features can be subjected to block fusion processing pm82 to obtain third-level image block features. The scale of the third-level image block features can be (H / 16)×(W / 16)×C. Next, the third-level image block features can be encoded en82 to obtain third-level processed features. It can be understood that the method of obtaining the third-level processed features based on the third-level image block features is the same or similar to the method of obtaining the processed features f41 based on the image block features pf40, and the present disclosure will not repeat them here. Through the embodiments of the present disclosure, the processed features at the previous level are subjected to block fusion processing, which can reduce the dimension of the features, reduce the computational complexity, and capture a wider range of contextual information, thereby improving the expressiveness of the global features.

[0102] In the disclosed embodiment, the coding feature is obtained based on at least one of the N-level processed features corresponding to the N-level image block features. For example, the third-level processed feature can be used as the coding feature.

[0103] In some embodiments, in some implementations of the above operation S240, decoding the coded features to obtain the processed image includes: performing N-level decoding on the coded features to obtain the processed image. Figure 8 As shown, the N-level decoding includes the first-level decoding de80, the second-level decoding de81 and the third-level decoding de82. After the N-level decoding of the encoded features, the processed image image81 can be obtained. Through the embodiment of the present disclosure, N-level encoding and N-level decoding are performed. Therefore, encoding and decoding are performed based on the U-net architecture, and a clear, high-resolution image can be reconstructed while retaining detail information.

[0104] It can be understood that the above describes the image processing method of the present disclosure. In some embodiments, the above image processing method can be implemented based on an image processing model, and the training method of the image processing model will be described below.

[0105] Fig. 9 is a flowchart of a method for training an image processing model according to another embodiment of the present disclosure.

[0106] In an embodiment of the present disclosure, the image processing model may include an encoding sub-model and a decoding sub-model, and the encoding sub-model includes an encoding network. The encoding network may include a first processing module, a scanning module, and an attention processing module.

[0107] like Fig. 9 As shown, the image processing model training method 900 may include operations S910 to S950.

[0108] In operation S910, the image block features of the sample image to be processed are input into a first processing module to obtain features to be processed corresponding to the image block features.

[0109] In the embodiment of the present disclosure, various processing can be performed on the image block features to determine the features to be processed. For example, the image block features can be used as the features to be processed.

[0110] In operation S920, the feature to be processed is input into a scanning module to obtain a plurality of scanning sequence features.

[0111] In an embodiment of the present disclosure, the scanning module corresponds to multiple scanning modes, and the features to be processed are features of multiple dimensions. The multiple scanning modes include a first scanning mode, and the first scanning mode can indicate scanning along a scanning direction, and the scanning direction is a direction that forms a scanning angle with a target dimension among the multiple dimensions of the features to be processed.

[0112] For example, the feature to be processed may be a feature of K dimensions. K may be an integer greater than 1. By scanning the feature of K dimensions, a scanning sequence feature of M dimensions may be obtained. M may be an integer less than K. Taking K=2 as an example, M may be 1. The scanning angle may be any angle value, such as an acute angle, an obtuse angle, a right angle, etc.

[0113] In operation S930, the plurality of scanning sequence features are input into an attention processing module to obtain processed features corresponding to the image block features.

[0114] In the disclosed embodiment, the attention processing module can process multiple scan sequence features based on the attention mechanism to determine multiple attention weights of the multiple scan sequence features. The multiple scan sequence features are weighted by using the multiple attention weights to obtain processed features. The weighted processing can be weighted summation, weighted average, etc.

[0115] In operation S940, the encoded features are input into a decoding sub-model to obtain a processed sample image.

[0116] In the disclosed embodiment, the coding feature may be obtained based on the processed feature. For example, the processed feature may be used as the coding feature.

[0117] In the embodiments of the present disclosure, the decoding sub-model may perform decoding based on various decoding methods, such as deconvolution, upsampling, etc.

[0118] In operation S950, an image processing model is trained based on the sample image and the processed sample image.

[0119] In the disclosed embodiment, a training loss may be determined based on the difference between the sample image and the processed sample image. The parameters of the image processing model are adjusted based on the training loss to train the parameters of the image processing model so that the image processed by the image processing model can be a high-quality, higher-definition or higher-resolution image.

[0120] It can be understood that the training method of the present invention is described above, and the image processing model of the present invention will be further described below.

[0121] Fig.10 is a schematic diagram of an image processing model according to an embodiment of the present disclosure.

[0122] like Fig.10As shown, the sample image image100 to be processed can be a foggy image. The above-mentioned image image30 to be processed can be used as the sample image image100 to be processed. The sample image image100 to be processed is input into the encoding sub-model en1000 to obtain the encoding feature. The encoding sub-module can, for example, implement the above-mentioned encoding process en30. The encoding feature is input into the decoding sub-model de1000 to obtain the processed sample image image101'. The decoding sub-model de1000 can implement the above-mentioned decoding process de30.

[0123] It can be understood that the above describes the image processing model of the present invention, and the encoding network of the present invention will be further described below.

[0124] In some embodiments, the encoding sub-model also includes an upsampling module. The above method may also include: inputting the sample image to be processed into the upsampling module to obtain the upsampled sample image. For example, the sample image to be processed may be pixel rearranged or interpolated to obtain the upsampled sample image. Through the embodiment of the present disclosure, the resolution of the sample image can be effectively improved. In addition, compared with interpolation processing, pixel rearrangement can better retain the detail information of the sample image and reduce information loss.

[0125] Next, in some embodiments, the encoding sub-model may further include a block processing module. The upsampled image may be input into the block processing module to obtain image block features of the sample image to be processed. For example, the block processing module may implement block embedding processing. The block embedding processing includes: determining multiple image blocks of the upsampled sample image. The size of the image block may be smaller than the size of the upsampled image. Multiple image blocks are linearly projected into a high-dimensional feature space to obtain image block features of the image to be processed. Thus, image information can be efficiently processed, and local structural features of the image can be captured, thereby improving the effect of subsequent processing. Next, the image block features may be processed to obtain processed features, which will be explained below.

[0126] Fig.11 is a schematic diagram of a coding network according to an embodiment of the present disclosure.

[0127] like Fig.11 As shown, the encoding network may include a first processing module c1100 and a visual state space subnetwork vss1100.

[0128] In some embodiments, the first processing module may include a plurality of convolutional layers and a first fusion layer. The plurality of convolutional layers correspond to a plurality of convolutional kernels respectively.

[0129] In some embodiments, in some implementations of the above operation S910, inputting the image block features of the sample image to be processed into the first processing module to obtain the features to be processed corresponding to the image block features includes: inputting the image block features into multiple convolution layers respectively to obtain multiple convolution results. Inputting the multiple convolution results into the first fusion layer to obtain the features to be processed. Fig.11 As shown, the first processing module may further include a preprocessing layer conv1100. The preprocessing layer conv1100 may correspond to a preprocessing convolution kernel. Before the image block features are respectively input into a plurality of convolution layers, the image block feature pf110 may be input into the preprocessing layer conv1100 to convolve the image block feature pf110 based on the preprocessing convolution kernel to obtain a preprocessing result. The preprocessing result is input into the first convolution layer conv1101 to obtain a first convolution result. The first convolution result is input into the batch normalization process bn1101 to obtain a normalized first convolution result. The preprocessing result is input into the convolution layer conv1102 to obtain a second convolution result. The second convolution result is input into the batch normalization process bn1102 to obtain a normalized second convolution result. The preprocessing result is input into the convolution layer conv1103 to obtain a third convolution result. The normalized first convolution result, the normalized second convolution result, and the third convolution result are input into the first fusion layer fuse1100 to obtain the features to be processed. The preprocessing convolution kernel may be a 1×1 convolution kernel. The first convolution kernel may be a 5×5 convolution kernel. The second convolution kernel may be a 3×3 convolution kernel. The third convolution kernel may be a 1×1 convolution kernel. Through the disclosed embodiment, based on multiple convolution kernels, local features of images of multiple scales may be obtained to make full use of features of different scales of image content. Thus, the local spatial information of the image may be better preserved, which helps to improve the image processing effect. Normalizing the convolution results can effectively improve the image processing efficiency, help to achieve convergence of the processing results, and improve the running speed of the model in the inference stage.

[0130] In some embodiments, the visual state space subnetwork may include the scanning module and the attention processing module described above. Fig.11 As shown, the feature to be processed is input into the visual state space sub-network vss1100 to obtain the visual state feature. According to the visual state feature, the processed feature can be obtained, which will be explained below.

[0131] In the disclosed embodiment, the encoding network may further include a second processing module. The second processing module may include a layer normalization layer and a multi-layer perceptron. The visual state feature is input into the layer normalization layer to obtain a normalized visual state feature. The normalized visual state feature is input into the multi-layer perceptron to obtain a multi-layer perceptual feature. The multi-layer perceptual feature is fused with the image block feature to obtain a processed feature. Fig.11 As shown, the visual state feature is input into the normalization layer ln1100 to obtain the normalized visual state feature. The normalized visual state feature is subjected to multi-layer perception processing mlp1100 to obtain the multi-layer perception feature. The multi-layer perception feature and the image block feature pf110 are input into the second fusion layer fuse1101 to obtain the processed feature f111.

[0132] It can be understood that the encoding network of the present disclosure is described above, and the visual state space subnetwork will be further described below.

[0133] In some embodiments, the visual state space subnetwork may include an adaptive selection scanning unit. The adaptive selection scanning unit may include the above-mentioned scanning module and the attention processing module.

[0134] In some embodiments, the above method may further include: processing the feature to be processed to obtain the feature to be scanned.

[0135] In the disclosed embodiment, the visual state space subnetwork also includes a first normalization layer, a first linear processing layer and a deep convolution layer. The features to be processed are input into the first normalization layer to obtain normalized features to be processed. The normalized features to be processed are input into the first linear processing layer to obtain the first linear features. The first linear features are input into the deep convolution layer to obtain the features to be scanned. The first normalization layer can, for example, implement the above-mentioned layer normalization processing ln51. The first linear processing layer can, for example, implement the above-mentioned linear processing l50. The deep convolution layer can, for example, implement the above-mentioned deep convolution depc50. The features to be scanned can be features of multiple dimensions, and the multiple dimensions of the features to be scanned can correspond to the multiple dimensions of the processing features. The dimension of the features to be scanned can also be K.

[0136] In some embodiments, in some implementations of the above operation S920, inputting the feature to be processed into a scanning module to obtain a plurality of scanning sequence features may include: scanning the feature to be scanned according to a plurality of scanning modes to obtain a plurality of scanning sequence features.

[0137] Next, the operation S930 may be performed. In some embodiments, in some embodiments of the operation S930, inputting multiple scan sequence features into the attention processing module to obtain processed features corresponding to the image block features may include: processing multiple scan sequence features based on the attention mechanism to obtain attention features. The attention feature may be, for example, the attention feature att50.

[0138] In some embodiments, the visual state space sub-network may further include a second normalization layer, a second linear processing layer, a third fusion layer, and a third linear processing layer.

[0139] In some embodiments, in some embodiments of the above operation S930, multiple scanning sequence features are input into the attention processing module to obtain processed features corresponding to the image block features, which may also include: inputting the attention features into the second normalization layer to obtain normalized attention features. Inputting the features to be processed into the second linear processing layer to obtain second linear features. Inputting the second linear features and the normalized attention features into the third fusion layer to obtain attention fusion features. Inputting the attention fusion features into the third linear processing layer to obtain visual state features. The second normalization processing layer can implement the above layer normalization processing ln52. The second linear processing layer can implement the above linear processing l51 to obtain the second linear feature. The third linear processing layer can implement the above linear processing l52. The visual state feature can be the above visual state feature vss40. It can be understood that the principle of the visual state space subnetwork can refer to Figure 5 The relevant descriptions of , will not be repeated in this disclosure.

[0140] It can be understood that the above describes how to obtain the visual state space subnetwork in the present disclosure, and some methods of the various scanning modes of the scanning module will be described below.

[0141] In some embodiments, the plurality of scanning modes may include a first scanning mode and a second scanning mode. The first scanning mode may indicate scanning along a scanning direction. The second scanning mode may indicate scanning based on a sliding window. According to the first scanning mode, the feature to be scanned is scanned to obtain a first scanning sequence feature. According to the second scanning mode, the feature to be scanned is scanned to obtain a second scanning sequence feature. It is understood that the description of the first scanning mode and the second scanning mode can refer to the above description. Fig. 6A The relevant description will not be repeated in this disclosure, and the processing method based on the attention mechanism will be described below.

[0142] In some embodiments, the scanning module can scan the feature to be scanned along different scanning directions to obtain multiple scanning sequence features. The scanning directions include from left to right, from top to bottom, from top left to bottom right, etc. The scanning module can also scan the feature to be scanned based on a 2×2 sliding window and a 3×3 sliding window to obtain multiple scanning sequence features.

[0143] In the embodiment of the present disclosure, the method may further include: inputting the plurality of scanning sequence features into a fourth fusion layer to obtain a scanning fusion feature. The fourth fusion layer may implement the fusion fuse 60 .

[0144] In an embodiment of the present disclosure, inputting multiple scan sequence features into an attention processing module to obtain processed features corresponding to image block features may include: inputting scan fusion features into multiple fully connected layers to obtain attention weight features. The dimension of the attention weight feature is consistent with the number of scan sequence features. The scan fusion features are weighted using the attention weight features to obtain attention features. For example, the attention mechanism may be an adaptive attention mechanism. The adaptive attention mechanism aatt60 may include performing multiple fully connected processing and weighted processing on the scan fusion features. Thus, by processing the scan fusion based on the adaptive attention mechanism, the attention features may be obtained. Based on the attention features, the processed features may be obtained.

[0145] It can be understood that the above describes the manner in which the present invention obtains attention features, and the adaptive attention mechanism of the present invention will be further described below.

[0146] In the disclosed embodiment, the attention processing module may include a pooling layer, a first fully connected layer, a first activation layer, a second fully connected layer, a second activation layer and a fifth fusion layer. Performing multiple fully connected processes on the scanned fusion features to obtain the attention weight features may include: inputting the scanned fusion features into the pooling layer to obtain the pooling features. Inputting the pooling features into the first fully connected layer to obtain the first fully connected features. Inputting the first fully connected features into the first activation layer to obtain the first activation features. Inputting the first activation features into the second fully connected layer to obtain the second fully connected features. Inputting the second fully connected features into the second activation layer to obtain the attention weight features. For example, the first pooling layer may implement the above-mentioned pooling process p70. The first fully connected layer may implement the above-mentioned first fully connected process fc70. The first activation layer may implement the above-mentioned first activation process relu70. The second fully connected layer may implement the second fully connected process fc71. The second activation layer may implement the second activation process sig70.

[0147] Next, the fifth fusion layer can use the attention weight feature to weight the scanned fusion feature to obtain the attention feature. The attention feature can be the above-mentioned attention feature att70. The processed feature can be decoded to obtain the processed image. It can be understood that the principle of the attention processing module can refer to Figure 7 The relevant descriptions of , will not be repeated in this disclosure.

[0148] It can be understood that the above takes the example of performing level 1 encoding and decoding on the sample image to be processed to illustrate the encoding and decoding methods of the present disclosure, but the present disclosure is not limited thereto, and the sample image to be processed can be encoded and decoded at N levels, which will be explained below.

[0149] Fig.12 is a schematic diagram of an image processing model according to another embodiment of the present disclosure.

[0150] like Fig.12 As shown, the image processing module may include an encoding sub-model en1200 and a decoding sub-model de1200. The encoding sub-model en1200 may include an N-level encoding network. The decoding sub-model may include an N-level decoding network.

[0151] In the embodiment of the present disclosure, the block processing module may include a block embedding processing module and a block fusion processing module. The above method may also include: inputting the sample image to be processed into the upsampling module to obtain the upsampled sample image. Inputting the upsampled image into the block embedding processing module to obtain the first level image block features of the sample image to be processed. Fig.12 As shown, the sample image image120 to be processed is input into the upsampling module u1200 to obtain the upsampled sample image. The upsampling module u1200 can, for example, implement the above-mentioned upsampling u80. The upsampled image input block is embedded in the processing module pe1200 to obtain the first-level image block features of the image to be processed. Next, the first-level image block features can be input into the first-level encoding network en1200 to obtain the first-level processed features. It can be understood that the method of obtaining the first-level processed features based on the first-level image block features is the same or similar to the method of obtaining the processed features f41 based on the image block features pf40, and the present disclosure will not be repeated here.

[0152] In the disclosed embodiment, the image processing model may further include a block fusion processing module. The nth level processed feature corresponding to the nth level image block feature is input into the block fusion processing module to obtain the n+1th level image block feature corresponding to the sample image to be processed. n is an integer greater than or equal to 1 and less than N, and N is an integer greater than 1.

[0153] like Fig.12 As shown, the first-level processed features can be input into the block fusion processing module pm1201 to obtain the second-level image block features. The second-level image block features can be input into the second-level encoding network en1201 to obtain the second-level processed features. It can be understood that the method of obtaining the second-level processed features according to the second-level image block features is the same or similar to the method of obtaining the processed features f41 according to the image block features pf40, and the present disclosure will not repeat them here.

[0154] like Figure 8 As shown, the second-level processed features can be input into the block fusion processing module pm1202 to obtain the third-level image block features. The third-level image block features can be input into the third-level encoding network en1202 to obtain the third-level processed features. It can be understood that the method of obtaining the third-level processed features according to the third-level image block features is the same or similar to the method of obtaining the processed features f41 according to the image block features pf40, and the present disclosure will not repeat them here.

[0155] In the disclosed embodiment, the coding feature is obtained based on at least one of the N-level processed features corresponding to the N-level image block features. For example, the third-level processed feature can be used as the coding feature.

[0156] In some embodiments, in some implementations of the above operation S940, inputting the encoded features into the decoding sub-model to obtain the processed sample image includes: inputting the encoded features into the N-level decoding network to obtain the processed image. Fig.12 As shown, the N-level decoding network includes a first-level decoding network de1200, a second-level decoding network de1201, and a third-level decoding network de1202. The encoding feature is input into the first-level decoding network de1200 to obtain the first-level decoding feature. The first-level decoding feature is input into the second-level decoding network de1201 to obtain the second-level decoding feature. The second-level decoding feature is input into the third-level decoding network de1202 to obtain the processed sample image image121.

[0157] It can be understood that the above describes the image processing model of the present invention, and some methods of training the model will be described below.

[0158] In some embodiments, in some implementations of the above operation S950, training the image processing model according to the sample image and the processed sample image includes: determining at least one of a mean absolute error loss and a structural loss according to the sample image and the processed sample image, and adjusting parameters of the image processing model according to at least one of the mean absolute error loss and the structural loss.

[0159] For example, based on multiple pixels of the label image of the sample image and multiple pixels of the processed sample image, the mean absolute error loss can be determined. Based on the sample image and the processed sample image, the structural loss can be determined. The training loss Loss can be determined by the following formula:

[0160] (Formula 1)

[0161] x can be the pixel value of the label image. i (y) can be the pixel value of the processed sample image. P is the number of pixels in the label image. i is an integer greater than or equal to 1 and less than or equal to P. λ can be a weight parameter. SSIM(·) is the structural similarity loss function (SSIM). The structural similarity loss function is an indicator to measure the similarity between two images. It takes into account the brightness, contrast and structure information, and quantifies the similarity between images by calculating the similarity between them.

[0162] In some embodiments, an adaptive moment estimation with weight decay (ADAM) optimizer can be used to train the image processing model. The first adaptability parameter (beta1) can be set to 0.9, the second adaptability parameter (beta2) can be set to 0.99, and the learning rate can be set to 1×10 -6 Each batch of training samples includes 256 sample images. The sample images can be cropped, randomly flipped horizontally or vertically, randomly rotated and scaled, and other image enhancement processes can be performed to further improve the generalization ability of the image processing model for different scenes. When the training loss is less than or equal to the preset loss threshold, the training can be stopped to obtain the trained image processing model.

[0163] It can be understood that the method of the present disclosure is described above, and the device of the present disclosure will be described below.

[0164] Fig.13 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0165] like Fig.13 As shown, the device 1300 may include a determination module 1310 , a first acquisition module 1320 , a second acquisition module 1330 and a decoding module 1340 .

[0166] The determination module 1310 is used to determine the to-be-processed feature corresponding to the image block feature according to the image block feature of the to-be-processed image.

[0167] The first acquisition module 1320 is used to obtain multiple scanning sequence features according to multiple scanning modes and features to be processed. The features to be processed are features of multiple dimensions, and the multiple scanning modes include a first scanning mode, which is used to indicate scanning along a scanning direction, and the scanning direction is a direction that forms a scanning angle with a target dimension in the multiple dimensions of the features to be processed.

[0168] The second acquisition module 1330 is used to process multiple scanning sequence features based on the attention mechanism to obtain processed features corresponding to the image block features.

[0169] The decoding module 1340 is used to decode the coded features to obtain a processed image. The coded features are obtained based on the processed features.

[0170] In some embodiments, the determination module includes: a first convolution submodule, configured to convolve the image block feature using at least one convolution kernel to obtain at least one convolution result; and a first acquisition submodule, configured to obtain a to-be-processed feature corresponding to the image block feature according to the at least one convolution result.

[0171] In some embodiments, the first acquisition module includes: a first processing submodule, which is used to process the feature to be processed to obtain the feature to be scanned. The feature to be scanned is a feature of multiple dimensions, and the multiple dimensions of the feature to be scanned correspond to the multiple dimensions of the processing feature. A first scanning submodule, which is used to scan the feature to be scanned according to multiple scanning modes to obtain multiple scanning sequence features.

[0172] In some embodiments, the first processing submodule includes: a first linear processing unit, configured to perform linear processing on the feature to be processed to obtain a first linear feature; and a first depth convolution unit, configured to perform depth convolution on the first linear feature to obtain a feature to be scanned.

[0173] In some embodiments, the multiple scanning modes further include a second scanning mode, the second scanning mode is used to indicate scanning based on a sliding window, and the first scanning submodule includes: a first scanning unit, used to scan the feature to be scanned according to the first scanning mode to obtain a first scanning sequence feature. A second scanning unit, used to scan the feature to be scanned according to the second scanning mode to obtain a second scanning sequence feature.

[0174] In some embodiments, there are multiple first scanning modes, and the multiple first scanning modes correspond to multiple different scanning angles. The value range of the scanning angle includes at least one of the following: greater than 0 degrees and less than 90 degrees, greater than 90 degrees and less than 180 degrees, greater than 180 degrees and less than 270 degrees, and greater than 270 degrees and less than 360 degrees.

[0175] In some embodiments, there are multiple second scanning modes, and the window parameters of the sliding windows of the multiple second scanning modes are different from each other, and the window parameter is at least one of the window size, the sliding direction and the sliding step size.

[0176] In some embodiments, the second acquisition module includes: a first fusion submodule, used to fuse multiple scan sequence features to obtain a scan fusion feature. A first fully connected submodule, used to perform multiple fully connected processes on the scan fusion feature to obtain an attention weight feature. The dimension of the attention weight feature is consistent with the number of scan sequence features. A first weighted submodule, used to weight the scan fusion feature using the attention weight feature to obtain an attention feature. A second acquisition submodule, used to obtain a processed feature based on the attention feature.

[0177] In some embodiments, the first fully connected submodule includes: a first pooling processing unit, used to perform pooling processing on the scan fusion feature to obtain a pooling feature. A first fully connected unit, used to perform a first fully connected processing on the pooled feature to obtain a first fully connected feature. A first activation unit, used to perform a first activation processing on the first fully connected feature to obtain a first activated feature. A second fully connected unit, used to perform a second fully connected processing on the first activated feature to obtain a second fully connected feature. A second activation unit, used to perform a second activation processing on the second fully connected feature to obtain an attention weight feature.

[0178] In some embodiments, the second obtaining submodule includes: a first obtaining unit, configured to obtain a visual state feature according to the attention feature and the feature to be processed; and a second obtaining unit, configured to obtain a processed feature according to the visual state feature.

[0179] In some embodiments, the first obtaining unit includes: a first normalization subunit, configured to perform normalization processing on the attention feature to obtain a normalized attention feature; a first linear processing subunit, configured to perform linear processing on the feature to be processed to obtain a second linear feature; and a first fusion subunit, configured to fuse the second linear feature with the normalized attention feature to obtain a visual state feature.

[0180] In some embodiments, the second obtaining unit includes: a second normalization subunit, configured to perform normalization processing on the visual state feature to obtain a normalized visual state feature; a first multi-layer perception processing unit, configured to perform multi-layer perception processing on the normalized visual state feature to obtain a multi-layer perception feature; and a second fusion subunit, configured to fuse the multi-layer perception feature with the image block feature to obtain a processed feature.

[0181] In some embodiments, the image block feature is one of the N-level image block features of the image to be processed, where N is an integer greater than or equal to 1. The apparatus further includes: a first upsampling module, configured to upsample the image to be processed to obtain an upsampled image; and a first block embedding module, configured to perform block embedding processing on the upsampled image to obtain a first-level image block feature of the image to be processed.

[0182] In some embodiments, the apparatus further comprises: a fusion module, configured to perform block fusion processing on the nth level processed features corresponding to the nth level image block features to obtain the n+1th level image block features corresponding to the image to be processed. n is an integer greater than or equal to 1 and less than N, and N is an integer greater than 1.

[0183] In some embodiments, the coding feature is obtained based on at least one of the N-level processed features corresponding to the N-level image block features. The decoding module is also used to perform N-level decoding on the coding feature to obtain a processed image.

[0184] In some embodiments, the image to be processed is an image to be defogged, and the processed image is a defogged image.

[0185] Fig.14 is a block diagram of a training device for an image processing model according to another embodiment of the present disclosure.

[0186] like Fig.14 As shown, the device 1400 may include a third acquisition module 1410, a fourth acquisition module 1420, a fifth acquisition module 1430, a sixth acquisition module 1440 and a training module 1450. The image processing model includes an encoding sub-model and a decoding sub-model, the encoding sub-model includes an encoding network, the encoding network includes a first processing module, a scanning module and an attention processing module,

[0187] The third obtaining module 1410 is used to input the image block features of the sample image to be processed into the first processing module to obtain the features to be processed corresponding to the image block features.

[0188] The fourth obtaining module 1420 is used to input the feature to be processed into the scanning module to obtain multiple scanning sequence features. The scanning module corresponds to multiple scanning modes, the feature to be processed is a feature of multiple dimensions, and the multiple scanning modes include a first scanning mode, and the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction that forms a scanning angle with a target dimension in the multiple dimensions of the feature to be processed.

[0189] The fifth acquisition module 1430 is used to input multiple scanning sequence features into the attention processing module to obtain processed features corresponding to the image block features.

[0190] The sixth obtaining module 1440 is used to input the coding feature into the decoding sub-model to obtain the processed sample image. The coding feature is obtained based on the processed feature.

[0191] The training module 1450 is used to train the image processing model according to the sample images and the processed sample images.

[0192] In some embodiments, the first processing module includes multiple convolution layers and a first fusion layer, and the multiple convolution layers correspond to multiple convolution kernels respectively. The third acquisition module includes: a third acquisition submodule, which is used to input the image block features into the multiple convolution layers respectively to obtain multiple convolution results. A fourth acquisition submodule is used to input the multiple convolution results into the first fusion layer to obtain the features to be processed.

[0193] In some embodiments, the fourth acquisition module includes: a second processing submodule, which is used to process the feature to be processed to obtain the feature to be scanned. The feature to be scanned is a feature of multiple dimensions, and the multiple dimensions of the feature to be scanned correspond to the multiple dimensions of the processing feature. A second scanning submodule is used to scan the feature to be scanned according to multiple scanning modes to obtain multiple scanning sequence features.

[0194] In some embodiments, the second processing submodule includes: a third obtaining unit, used to input the feature to be processed into the first normalization layer to obtain the normalized feature to be processed. A fourth obtaining unit, used to input the normalized feature to be processed into the first linear processing layer to obtain the first linear feature. A fifth obtaining unit, used to input the first linear feature into the deep convolution layer to obtain the feature to be scanned.

[0195] In some embodiments, the plurality of scanning modes further include a second scanning mode, and the second scanning mode is used to indicate scanning based on a sliding window. The second scanning submodule includes: a third scanning unit, used to scan the feature to be scanned according to the first scanning mode to obtain a first scanning sequence feature. A fourth scanning unit, used to scan the feature to be scanned according to the second scanning mode to obtain a second scanning sequence feature.

[0196] In some embodiments, there are multiple first scanning modes, and the multiple first scanning modes correspond to multiple different scanning angles. The value range of the scanning angle includes at least one of the following: greater than 0 degrees and less than 90 degrees, greater than 90 degrees and less than 180 degrees, greater than 180 degrees and less than 270 degrees, and greater than 270 degrees and less than 360 degrees.

[0197] In some embodiments, there are multiple second scanning modes, and the window parameters of the sliding windows of the multiple second scanning modes are different from each other, and the window parameter is at least one of the window size, the sliding direction and the sliding step size.

[0198] In some embodiments, the above-mentioned device also includes: a fifth acquisition submodule, which is used to input multiple scanning sequence features into the fourth fusion layer to obtain scanning fusion features. The fifth acquisition module includes: a sixth acquisition submodule, which is used to input the scanning fusion features into multiple fully connected layers to obtain attention weight features. The dimension of the attention weight feature is consistent with the number of scanning sequence features. A second weighting submodule is used to weight the scanning fusion features using the attention weight features to obtain attention features. A seventh acquisition submodule is used to obtain processed features based on the attention features.

[0199] In some embodiments, the sixth acquisition submodule includes: a sixth acquisition unit, used to input the scan fusion feature into the pooling layer to obtain the pooling feature. A seventh acquisition unit, used to input the pooling feature into the first fully connected layer to obtain the first fully connected feature. An eighth acquisition unit, used to input the first fully connected feature into the first activation layer to obtain the first activation feature. A ninth acquisition unit, used to input the first activation feature into the second fully connected layer to obtain the second fully connected feature. A tenth acquisition unit, used to input the second fully connected feature into the second activation layer to obtain the attention weight feature.

[0200] In some embodiments, the seventh obtaining submodule includes: an eleventh obtaining unit, configured to obtain a visual state feature according to the feature to be processed and the attention feature; and a twelfth obtaining unit, configured to obtain a processed feature according to the visual state feature.

[0201] In some embodiments, the eleventh obtaining unit includes: a first obtaining subunit, used to input the attention feature into the second normalization layer to obtain the normalized attention feature. A second obtaining subunit, used to input the feature to be processed into the second linear processing layer to obtain the second linear feature. A third obtaining subunit, used to input the second linear feature and the normalized attention feature into the third fusion layer to obtain the attention fusion feature. A fourth obtaining subunit, used to input the attention fusion feature into the third linear processing layer to obtain the visual state feature.

[0202] In some embodiments, the twelfth obtaining unit includes: a fifth obtaining unit, configured to input the visual state feature into a fifth normalization layer to obtain a normalized visual state feature. A sixth obtaining unit, configured to input the normalized visual state feature into a multi-layer perceptron to obtain a multi-layer perceptual feature. A seventh obtaining unit, configured to fuse the multi-layer perceptual feature with the image block feature to obtain a processed feature.

[0203] In some embodiments, the image block feature is one of the N-level image block features of the image to be processed, where N is an integer greater than or equal to 1. The above-mentioned apparatus further includes: a seventh obtaining module, which is used to input the sample image to be processed into the upsampling module to obtain the upsampled sample image. An eighth obtaining module, which is used to embed the upsampled image input block into the processing module to obtain the first-level image block feature of the sample image to be processed.

[0204] In some embodiments, the apparatus further includes: a ninth acquisition module, configured to input the nth level processed feature corresponding to the nth level image block feature into the nth level block fusion process to obtain the n+1th level image block feature of the image to be processed. n is an integer greater than or equal to 1 and less than N, and N is an integer greater than 1.

[0205] In some embodiments, the coding feature is obtained based on at least one of the N-level processed features corresponding to the N-level image block features. The sixth obtaining module is further used to: input the coding feature into the N-level decoding network to obtain the processed image.

[0206] In some embodiments, the image to be processed is an image to be defogged, and the processed image is a defogged image.

[0207] In some embodiments, the training module includes: a first determination submodule, configured to determine at least one of a mean absolute error loss and a structural loss according to the sample image and the processed sample image; and an adjustment submodule, configured to adjust parameters of the image processing model according to at least one of the mean absolute error loss and the structural loss.

[0208] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0209] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0210] Fig.15 A schematic block diagram of an example electronic device 1500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0211] like Fig.15 As shown, the device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1502 or a computer program loaded from a storage unit 1508 to a random access memory (RAM) 1503. In the RAM 1503, various programs and data required for the operation of the device 1500 can also be stored. The computing unit 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0212] A number of components in the device 1500 are connected to the I / O interface 1505, including: an input unit 1506, such as a keyboard, a mouse, etc.; an output unit 1507, such as various types of displays, speakers, etc.; a storage unit 1508, such as a disk, an optical disk, etc.; and a communication unit 1509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1509 allows the device 1500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0213] The computing unit 1501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1501 performs the various methods and processes described above, such as image processing methods and / or training methods for image processing models. For example, in some embodiments, the image processing methods and / or training methods for image processing models may be implemented as computer software programs, which are tangibly contained in machine-readable media, such as storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1500 via ROM 1502 and / or communication unit 1509. When the computer program is loaded into RAM 1503 and executed by the computing unit 1501, one or more steps of the image processing method and / or the training method of the image processing model described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured to perform the image processing method and / or the training method of the image processing model by any other appropriate means (e.g., by means of firmware).

[0214] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0215] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0216] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0217] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0218] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0219] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0220] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0221] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: According to the image block features of the image to be processed, determining the features to be processed corresponding to the image block features; According to the plurality of scanning modes and the features to be processed, a plurality of scanning sequence features are obtained, wherein the features to be processed are features of multiple dimensions, and the plurality of scanning modes include a first scanning mode, the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction forming a scanning angle with a target dimension among the multiple dimensions of the features to be processed; Processing the plurality of scanning sequence features based on an attention mechanism to obtain processed features corresponding to the image block features; The coded features are decoded to obtain a processed image, wherein the coded features are obtained based on the processed features.

2. The method according to claim 1, wherein: The determining, according to the image block feature of the image to be processed, the feature to be processed corresponding to the image block feature comprises: Using at least one convolution kernel, convolve the image block features to obtain at least one convolution result; According to at least one of the convolution results, the feature to be processed corresponding to the feature of the image block is obtained.

3. The method according to claim 1, wherein: The obtaining of multiple scanning sequence features according to the multiple scanning modes and the features to be processed includes: Processing the feature to be processed to obtain a feature to be scanned, wherein the feature to be scanned is a feature of multiple dimensions, and the multiple dimensions of the feature to be scanned correspond to the multiple dimensions of the processing feature; The features to be scanned are scanned according to the plurality of scanning modes to obtain a plurality of scanning sequence features.

4. The method according to claim 3, wherein: The processing of the feature to be processed to obtain the feature to be scanned comprises: Performing linear processing on the feature to be processed to obtain a first linear feature; Performing depth convolution on the first linear feature to obtain the feature to be scanned.

5. The method according to claim 3, wherein: The plurality of scanning modes further include a second scanning mode, wherein the second scanning mode is used to indicate scanning based on a sliding window, The scanning of the to-be-scanned features according to the plurality of scanning modes to obtain the plurality of scanning sequence features comprises: Scanning the feature to be scanned according to the first scanning mode to obtain a first scanning sequence feature; According to the second scanning mode, the feature to be scanned is scanned to obtain a second scanning sequence feature.

6. The method according to claim 1 or 5, wherein: There are multiple first scanning modes, and the multiple first scanning modes correspond to multiple different scanning angles. The value range of the scanning angle includes at least one of the following: greater than 0 degrees and less than 90 degrees, greater than 90 degrees and less than 180 degrees, greater than 180 degrees and less than 270 degrees, and greater than 270 degrees and less than 360 degrees.

7. The method according to claim 5, wherein: There are multiple second scanning modes, and window parameters of the sliding windows of the multiple second scanning modes are different from each other. The window parameter is at least one of a window size, a sliding direction, and a sliding step size.

8. The method according to claim 1, wherein: The processing of the plurality of scanning sequence features based on the attention mechanism to obtain the processed features corresponding to the image block features comprises: Fusing a plurality of the scanning sequence features to obtain a scanning fusion feature; Performing multiple full-connection processing on the scan fusion features to obtain attention weight features, wherein the dimension of the attention weight features is consistent with the number of the scan sequence features; The scan fusion feature is weighted by using the attention weight feature to obtain an attention feature; The processed feature is obtained according to the attention feature.

9. The method according to claim 8, wherein: The scanning fusion feature is subjected to multiple full connection processing to obtain the attention weight feature, which includes: Performing pooling processing on the scanned fusion features to obtain pooled features; Performing a first fully connected process on the pooled feature to obtain a first fully connected feature; Performing a first activation process on the first fully connected feature to obtain a first activated feature; Performing a second fully connected process on the first activated feature to obtain a second fully connected feature; Perform a second activation process on the second fully connected feature to obtain the attention weight feature.

10. The method according to claim 8, wherein: The obtaining the processed feature according to the attention feature comprises: Obtaining a visual state feature according to the attention feature and the feature to be processed; The processed feature is obtained according to the visual state feature.

11. The method according to claim 10, wherein: The obtaining of the visual state feature according to the attention feature and the feature to be processed comprises: Normalizing the attention feature to obtain a normalized attention feature; Performing linear processing on the feature to be processed to obtain a second linear feature; The second linear feature and the normalized attention feature are fused to obtain the visual state feature.

12. The method according to claim 10, wherein: The obtaining the processed feature according to the visual state feature comprises: Normalizing the visual state feature to obtain a normalized visual state feature; Performing multi-layer perception processing on the normalized visual state features to obtain multi-layer perception features; The multi-layer perception feature and the image block feature are fused to obtain the processed feature.

13. The method according to claim 1, wherein: The image block feature is one of the N-level image block features of the image to be processed, where N is an integer greater than or equal to 1. Also includes: Upsampling the image to be processed to obtain an upsampled image; The upsampled image is subjected to block embedding processing to obtain first-level image block features of the image to be processed.

14. The method according to claim 13, further comprising: Block fusion processing is performed on the nth level processed features corresponding to the nth level image block features to obtain the n+1th level image block features of the image to be processed, where n is an integer greater than or equal to 1 and less than N, and N is an integer greater than 1.

15. The method according to claim 13, wherein: The coding feature is obtained based on at least one of the N-level processed features corresponding to the N-level image block features; The decoding of the coded features to obtain a processed image comprises: The encoded features are decoded in N stages to obtain the processed image.

16. The method according to claim 1, wherein: The image to be processed is an image to be defogged, and the processed image is a defogged image.

17. A training method for an image processing model, the image processing model comprising an encoding sub-model and a decoding sub-model, the encoding sub-model comprising an encoding network, the encoding network comprising a first processing module, a scanning module and an attention processing module, the method comprising: Inputting the image block features of the sample image to be processed into the first processing module to obtain the features to be processed corresponding to the image block features; Input the feature to be processed into the scanning module to obtain a plurality of scanning sequence features, wherein the scanning module corresponds to a plurality of scanning modes, the feature to be processed is a feature of a plurality of dimensions, and the plurality of scanning modes include a first scanning mode, the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction forming a scanning angle with a target dimension among the plurality of dimensions of the feature to be processed; Inputting a plurality of the scanning sequence features into the attention processing module to obtain processed features corresponding to the image block features; Inputting the encoding features into a decoding sub-model to obtain a processed sample image, wherein the encoding features are obtained based on the processed features; The image processing model is trained according to the sample image and the processed sample image.

18. The method according to claim 17, wherein: The first processing module includes a plurality of convolutional layers and a first fusion layer, wherein the plurality of convolutional layers correspond to a plurality of convolutional kernels respectively. The step of inputting the image block features of the sample image to be processed into the first processing module to obtain the features to be processed corresponding to the image block features comprises: Inputting the image block features into the multiple convolution layers respectively to obtain multiple convolution results; Input the multiple convolution results into the first fusion layer to obtain the features to be processed.

19. The method according to claim 17, wherein: The training of the image processing model according to the sample image and the processed sample image comprises: Determine at least one of a mean absolute error loss and a structural loss according to the sample image and the processed sample image; Parameters of the image processing model are adjusted according to at least one of the mean absolute error loss and the structural loss.

20. An image processing device, comprising: A determination module, used to determine, based on the image block features of the image to be processed, features to be processed corresponding to the image block features; A first obtaining module is used to obtain a plurality of scanning sequence features according to a plurality of scanning modes and the features to be processed, wherein the features to be processed are features of multiple dimensions, the plurality of scanning modes include a first scanning mode, the first scanning mode is used to indicate scanning along a scanning direction, and the scanning direction is a direction forming a scanning angle with a target dimension among the multiple dimensions of the features to be processed; A second acquisition module is used to process the plurality of scanning sequence features based on an attention mechanism to obtain processed features corresponding to the image block features; A decoding module is used to decode the coded features to obtain a processed image, wherein the coded features are obtained based on the processed features.

21. A training device for an image processing model, the image processing model comprising an encoding sub-model and a decoding sub-model, the encoding sub-model comprising an encoding network, the encoding network comprising a first processing module, a scanning module and an attention processing module, the device comprising: A third obtaining module, used for inputting the image block features of the sample image to be processed into the first processing module to obtain the features to be processed corresponding to the image block features; a fourth obtaining module, configured to input the feature to be processed into the scanning module to obtain a plurality of scanning sequence features, wherein the scanning module corresponds to a plurality of scanning modes, the feature to be processed is a feature of a plurality of dimensions, the plurality of scanning modes include a first scanning mode, the first scanning mode is configured to indicate scanning along a scanning direction, the scanning direction being a direction forming a scanning angle with a target dimension among the plurality of dimensions of the feature to be processed; A fifth obtaining module, configured to input the plurality of scanning sequence features into the attention processing module to obtain processed features corresponding to the image block features; a sixth obtaining module, configured to input the coding feature into a decoding sub-model to obtain a processed sample image, wherein the coding feature is obtained based on the processed feature; A training module is used to train the image processing model according to the sample image and the processed sample image.

22. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 19.

23. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 19.

24. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 19.

Images