An image processing method, apparatus, device and medium

Abstract

This invention discloses an image processing method, apparatus, device, and medium. In this invention, a target image to be processed is acquired, and a shallow feature map is obtained by convolutional processing of the input target image based on the convolutional layers of a pre-trained hybrid attention super-resolution network model. The shallow feature map is then sequentially processed through each attention residual layer in a predetermined number of cascaded attention residual layers. Because the predetermined number of cascaded attention residual layers constructs a deeper network, the model pays more attention to the detailed regions in the image. The deeper network can accurately extract deep feature maps of detailed features. The pixel values ​​of the corresponding rows and columns of the deep feature map and the shallow feature map are added together and then input into an upsampling layer to obtain a high-resolution image with enhanced details. This solves the problem of inaccurate feature extraction of blurred detailed regions in thermal imaging images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and medium. Background Technology

[0002] Due to hardware and cost limitations, thermal imaging images typically have low resolution and lack detail. Super-resolution reconstruction can improve image resolution and quality, alleviating the problems of low resolution and lack of detail.

[0003] However, thermal imaging images are less detailed than visible light images, making it difficult to distinguish between detailed and flat areas. Furthermore, when there is a significant temperature difference between the target and the surrounding environment, feature extraction of detailed areas becomes even less accurate. After thermal imaging images are super-resolved using a super-resolution network model, black and white edges appear in the detailed areas.

[0004] Therefore, how to solve the black and white edge phenomenon in super-resolution reconstruction caused by inaccurate feature extraction of blurred detail areas in thermal imaging images has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, device, and medium to solve the problem of black and white edges in super-resolution reconstruction caused by inaccurate feature extraction of blurred detail areas in thermal imaging images in the prior art.

[0006] This invention provides an image processing method, the method comprising:

[0007] The target image to be processed is obtained, and a shallow feature map is obtained by performing convolution processing on the input target image based on the convolutional layer of the pre-trained hybrid attention super-resolution network model.

[0008] Based on a predetermined number of cascaded attention residual layers in the hybrid attention super-resolution network model, the shallow feature map is sequentially processed through each attention residual layer to obtain the deep feature map output by the last attention residual layer. Based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of the corresponding rows and columns of the shallow feature map and the deep feature map are added to obtain the processed target feature map. Based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain the output high-resolution image with enhanced details.

[0009] Furthermore, the deep feature map output by the last attention residual layer, obtained by sequentially processing the shallow feature map through each attention residual layer in the hybrid attention super-resolution network model with a predetermined number of cascaded attention residual layers, includes:

[0010] For each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map is used as the first input feature map and input into the attention residual layer. If the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is used as the first input feature map of the attention residual layer. The attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing. Based on the residual addition processing layer of the attention residual layer, the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map are added to obtain the output feature map of the attention residual layer, until the deep feature map output by the last attention residual layer is obtained.

[0011] Furthermore, the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing, including:

[0012] The processing unit of the attention layer based on the attention residual layer inputs the pixel value of each pixel in the first input feature map and the local binary pattern LBP sampling function pre-stored in the attention layer to obtain the output LBP feature value matrix and stores it in the attention layer.

[0013] The first input feature map is input into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map. The LBP feature value matrix is ​​then processed by the second perceptron unit of the attention layer to obtain a gradient-based attention feature map. The two attention feature maps are then input into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map. Finally, the fused attention feature map is multiplied by the first input feature map by the dot product processing unit of the attention layer to obtain the output target attention feature map.

[0014] Furthermore, each attention residual layer further includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. Before the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the perceptually processed attention layer, the method further includes:

[0015] The first input feature map is input into the first convolutional layer of the attention residual layer for convolution processing, then through the first activation layer for activation processing, then through the second convolutional layer for convolution processing, and then through the second activation layer for activation processing to obtain the second input feature map. The second input feature map is then input into the attention layer of the attention residual layer for subsequent processing.

[0016] Accordingly, the present invention provides an image processing apparatus, the apparatus comprising:

[0017] The acquisition module is used to acquire the target image to be processed;

[0018] The processing module is used to perform convolution processing on the input target image based on the convolutional layers of a pre-trained hybrid attention super-resolution network model to obtain a shallow feature map; based on a predetermined number of cascaded attention residual layers in the hybrid attention super-resolution network model, the shallow feature map is sequentially passed through each attention residual layer for perceptual processing to obtain a deep feature map output by the last attention residual layer; based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of the corresponding rows and columns of the shallow feature map and the deep feature map are added to obtain the processed target feature map; based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain an output high-resolution image with enhanced details.

[0019] Further, the processing module is specifically configured to, for each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, input the shallow feature map as the first input feature map into the attention residual layer; if the attention residual layer is not the first attention residual layer, use the output feature map of the previous attention residual layer as the first input feature map of the attention residual layer; perform perceptual processing on the first input feature map based on the attention layer of the attention residual layer to obtain the target attention feature map output by the attention layer after perceptual processing; and add the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map based on the residual addition processing layer of the attention residual layer to obtain the output feature map of the attention residual layer, until the deep feature map output by the last attention residual layer is obtained.

[0020] Further, the processing module is specifically configured to: a processing unit of the attention layer based on the attention residual layer; input the pixel value of each pixel in the first input feature map and the pre-stored Local Binary Pattern (LBP) sampling function in the attention layer to obtain an output LBP feature value matrix and store it in the attention layer; input the first input feature map into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map; process the LBP feature value matrix through the second perceptron unit of the attention layer to obtain a gradient-based attention feature map; input the two attention feature maps into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map; and, based on the dot product processing unit of the attention layer, perform a dot product processing on the fused attention feature map and the first input feature map to obtain the output target attention feature map.

[0021] Furthermore, each attention residual layer also includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. The processing module is specifically used to input the first input feature map into the first convolutional layer of the attention residual layer for convolution processing, through the first activation layer for activation processing, through the second convolutional layer for convolution processing, through the second activation layer for activation processing to obtain a second input feature map, and input the second input feature map into the attention layer of the attention residual layer for subsequent processing.

[0022] Accordingly, the present invention provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0023] The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of any of the above-described image processing methods.

[0024] Accordingly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described image processing methods.

[0025] This invention provides an image processing method, apparatus, device, and medium. In this invention, the target image to be processed is acquired, and a shallow feature map is obtained by convolutional processing of the input target image based on the convolutional layers of a pre-trained hybrid attention super-resolution network model. The shallow feature map is then processed sequentially through each attention residual layer using a predetermined number of cascaded attention residual layers. Because the predetermined number of cascaded attention residual layers constructs a deeper network, the model pays more attention to the detailed regions in the image. The deeper network can accurately extract deep feature maps of detailed features. The pixel values ​​of the corresponding rows and columns of the deep feature map and the shallow feature map are added together and then input into an upsampling layer to obtain a high-resolution image with enhanced details. This solves the problem of inaccurate feature extraction of blurred detailed regions in thermal imaging images. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an image processing method provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of an image processing procedure provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram illustrating the process by which the attention layer of a hybrid attention neural network model, provided in an embodiment of the present invention, performs perceptual processing on a first input feature map.

[0030] Figure 4 This is a schematic diagram illustrating the process of extracting output feature maps using an attention residual layer, as provided in an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] To address the issue of black and white borders in super-resolution reconstruction caused by inaccurate feature extraction of blurred detail areas in thermal imaging images, embodiments of the present invention provide an image processing method, apparatus, device, and medium.

[0035] Example 1:

[0036] Figure 1 This is a schematic diagram of an image processing method provided in an embodiment of the present invention. The process includes the following steps:

[0037] S101: Obtain the target image to be processed, and perform convolution processing on the input target image based on the convolutional layer of the pre-trained hybrid attention super-resolution network model to obtain a shallow feature map.

[0038] To address the issue of black and white borders in super-resolution reconstruction caused by inaccurate feature extraction of blurred detail areas in thermal imaging images, this invention provides an image processing method applied to electronic devices. These electronic devices can be smart terminal devices such as host computers, tablets, and smartphones, or servers. The server can be a local server or a cloud server, and this invention does not impose any limitations on this.

[0039] The electronic device acquires a target image to be processed, which can be a thermal imaging image or a low-resolution image with blurred details, such as an infrared image or a visible light image. Specifically, the electronic device can receive a target image sent by an electronic device connected to it, or it can acquire a target image stored in the electronic device itself.

[0040] To extract shallow feature maps of the target image, the electronic device stores a pre-trained hybrid attention super-resolution network model. This hybrid attention super-resolution network model is used to achieve super-resolution reconstruction of low-resolution images. The loss function of the model is L = MSE(lr, hr), where L represents the loss function value, lr represents the low-resolution image, hr represents the high-resolution image, and MSE is the mean squared error. Based on the convolutional layers of this hybrid attention super-resolution network model, the target image is convolved to obtain the shallow feature map of the target image.

[0041] S102: Based on a set number of cascaded attention residual layers in the hybrid attention super-resolution network model, the shallow feature map is sequentially processed through each attention residual layer to obtain the deep feature map output by the last attention residual layer. Based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of the corresponding rows and columns of the shallow feature map and the deep feature map are added to obtain the processed target feature map. Based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain the output high-resolution image with enhanced details.

[0042] In order to extract deep feature maps of the target image, the hybrid attention super-resolution network model includes a set number of cascaded attention residual layers after the convolutional layers. The set number can be 60, 64, 62, 65, etc., and preferably 64.

[0043] The electronic device inputs the shallow feature map output from the convolutional layer into the first attention residual layer, sequentially passes it through each attention residual layer for perceptual processing, uses the output feature map of the previous attention residual layer as the input feature map of the next attention residual layer, and obtains the deep feature map output by the last attention residual layer.

[0044] Figure 2 This is a schematic diagram of an image processing process provided in an embodiment of the present invention, such as... Figure 2 As shown, a low-resolution image is input into the convolutional layer of a hybrid attention neural network model, and the shallow feature map output by the convolutional layer is input into the first attention residual layer. After perceptual processing by 64 attention residual layers, the deep feature map output by the last attention residual layer is obtained. The deep feature map and the shallow feature map are input into the target residual addition processing layer to obtain the target feature map output by the target residual addition processing layer, which is then input into the upsampling layer to obtain the high-resolution image output by the upsampling layer.

[0045] The target residual summation layer of the hybrid attention super-resolution network model inputs the shallow and deep feature maps into the target residual summation layer. Based on the pixel values ​​of corresponding rows and columns in the shallow and deep feature maps, the pixel values ​​of corresponding rows and columns are summed to obtain the target feature map after residual summation. The upsampling layer of the hybrid attention super-resolution network model inputs the target feature map into the upsampling layer to perform upsampling processing on the target image, resulting in a high-resolution image with enhanced details output by the upsampling layer.

[0046] In this embodiment of the invention, the target image to be processed is acquired, and a shallow feature map is obtained by convolutional processing of the input target image based on the convolutional layers of a pre-trained hybrid attention super-resolution network model. The shallow feature map is then processed sequentially through each attention residual layer by a set number of cascaded attention residual layers in the model. Since the set number of cascaded attention residual layers constructs a deeper network, the model pays more attention to the detailed regions in the image. The deeper network can accurately extract the deep feature map of the detailed features. The pixel values ​​of the corresponding rows and columns of the deep feature map and the shallow feature map are added together and then input into the upsampling layer to obtain the high-resolution image with enhanced details. This solves the problem of inaccurate feature extraction of blurred detailed regions in thermal imaging images.

[0047] Example 2:

[0048] To obtain a deep feature map, based on the above embodiments, in this embodiment of the invention, the shallow feature map is sequentially processed through each attention residual layer in the hybrid attention super-resolution network model using a predetermined number of cascaded attention residual layers to obtain the deep feature map output by the last attention residual layer, including:

[0049] For each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map is used as the first input feature map and input into the attention residual layer. If the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is used as the first input feature map of the attention residual layer. The attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing. Based on the residual addition processing layer of the attention residual layer, the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map are added to obtain the output feature map of the attention residual layer, until the deep feature map output by the last attention residual layer is obtained.

[0050] For each attention residual layer of the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map output by the convolutional layer is used as the first input feature map and input into the attention residual layer; if the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is obtained and used as the first input feature map of the attention residual layer.

[0051] The attention residual layer includes an attention layer and a residual addition processing layer. A first input feature map is input into the attention layer of the attention residual layer. Based on the attention layer, the first input feature map is subjected to perceptual processing to obtain a target attention feature map output by the attention layer after perceptual processing. The target attention feature map and the first input feature map are input into the residual addition processing layer of the attention residual layer. Based on the target attention feature map and the first input feature map input into the residual addition processing layer of the attention residual layer, the pixel value of each pixel in the target attention feature map and the first input feature map is determined. The pixel values ​​of the corresponding rows and columns in the target attention feature map and the first input feature map are added to obtain the output feature map of the attention residual layer. If the attention residual layer is not the last attention residual layer, the output feature map is used as the input feature map of the next attention residual layer. If the attention residual layer is the last attention residual layer, the output feature map is used as the deep feature map output by the last attention residual layer.

[0052] Example 3:

[0053] To obtain the target attention feature map, based on the above embodiments, in this embodiment of the invention, the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing, including:

[0054] The processing unit of the attention layer based on the attention residual layer inputs the pixel value of each pixel in the first input feature map and the local binary pattern (LBP) sampling function pre-stored in the attention layer to obtain the output LBP feature value matrix and stores it in the attention layer.

[0055] The first input feature map is input into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map. The LBP feature value matrix is ​​then processed by the second perceptron unit of the attention layer to obtain a gradient-based attention feature map. The two attention feature maps are then input into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map. Finally, the fused attention feature map is multiplied by the first input feature map by the dot product processing unit of the attention layer to obtain the output target attention feature map.

[0056] To obtain the target attention feature map, after inputting the first input feature map into the attention layer of the attention residual layer, the first input feature map is first input into the processing unit of the attention layer. Based on the pixel value of each pixel in the first input feature map and the pre-saved LBP sampling function, the pixel value of each pixel is input into the LBP sampling function to obtain the LBP feature value corresponding to each pixel. Based on the LBP feature value corresponding to each pixel and the row and column where each pixel is located in the first input feature map, the LBP feature value corresponding to each pixel is used as the element value of the corresponding row and column in the LBP feature value matrix, thereby obtaining the LBP feature value matrix and storing it in the attention layer.

[0057] The first input feature map is input into the first perceptron unit of the attention layer, and the first input feature map is processed to obtain a brightness-based attention feature map; the LBP feature value matrix is ​​input into the second perceptron unit of the attention layer, and the LBP feature value matrix is ​​processed to obtain a gradient-based attention feature map, wherein the first perceptron unit and the second perceptron unit are multi-layered and independent of each other.

[0058] The brightness-based attention feature map and the gradient-based attention feature map are input into the fusion layer unit of the attention layer. The brightness-based attention feature map and the gradient-based attention feature map are fused to obtain the fused attention feature map. The fused attention feature map and the first input feature map are input into the dot product processing unit of the attention layer. The fused attention feature map and the first input feature map are multiplied by the pixel value of each row and column of the first input feature map according to the pixel value of each position of the first input feature map, so as to obtain the pixel value of each row and column of the target attention feature map.

[0059] Figure 3 This is a schematic diagram illustrating the process by which the attention layer of a hybrid attention neural network model, according to an embodiment of the present invention, performs perceptual processing on a first input feature map. Figure 3 As shown, the first input feature map is fed into the fusion layer after passing through the first multilayer perceptron. The first input feature map is then fed into the LBP feature value matrix after passing through the determination unit. The LBP feature value matrix is ​​then fed into the fusion layer after passing through the second multilayer perceptron. The fused feature map output by the fusion layer is multiplied by the first input feature map to obtain the output target attention feature map.

[0060] As a possible implementation, in this embodiment of the invention, after acquiring the target image, the electronic device can obtain the output LBP feature value matrix based on the target image and the pre-saved LBP sampling function, and store it in the attention layer of each attention residual layer.

[0061] The following specific embodiment illustrates the process by which the attention layer of the present invention performs perceptual processing on the first input feature map, based on the target image I and the LBP sampling function F. s The pixel value of each pixel in the target image is input into the LBP sampling function to obtain the LBP eigenvalue matrix Fea. Lbp Fea Lbp =F s (I) The first input feature map and the LBP eigenvalue matrix are input into two independent multilayer perceptrons respectively. After processing by the multilayer perceptrons, the output is a brightness-based attention feature map Att. Luma and gradient-based attention feature map Att Lbp Att Luma =MLP Luma (Fea Luma ), Att Lbp =MLP Lbpa (Fea Lbp MLP Luma and MLP Lbpa These represent two independent perceptrons, Fea and Fea. Luma This represents the first input feature map.

[0062] The two attention feature maps are fused in the fusion layer to obtain the fused attention feature map Att. Final Att Final =F Fus (Att Lbp +Att Luma ), where F Fus The fusion layer is represented by the dot product of the fused attention feature map and the first input feature map, resulting in the target attention feature map with the added attention feature map.

[0063] Example 4:

[0064] To improve the accuracy of feature extraction, based on the above embodiments, in this embodiment of the invention, each attention residual layer further includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. Before the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the perceptually processed attention layer, the method further includes:

[0065] The first input feature map is input into the first convolutional layer of the attention residual layer for convolution processing, then through the first activation layer for activation processing, then through the second convolutional layer for convolution processing, and then through the second activation layer for activation processing to obtain the second input feature map. The second input feature map is then input into the attention layer of the attention residual layer for subsequent processing.

[0066] To improve the accuracy of feature extraction, each attention residual layer also includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. After the first input feature map is input into the attention residual layer, the first input feature map is convolved through the first convolutional layer. The convolved first input feature map is then input into the first activation layer for activation. The activated first input feature map is then input into the second convolutional layer for convolution. Finally, the convolved first input feature map is input into the second activation layer for activation, resulting in the second input feature map output by the second activation layer.

[0067] The second input feature map is input into the attention layer of the attention residual layer for perceptual processing to obtain the target attention feature map output by the attention layer after perceptual processing. Based on the residual addition processing layer of the attention residual layer, the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map are added together to obtain the output feature map of the attention residual layer.

[0068] Figure 4 This is a schematic diagram illustrating the process of extracting output feature maps using an attention residual layer, as provided in an embodiment of the present invention. Figure 4 As shown, the first input feature map is passed through a convolutional layer, an activation layer, another convolutional layer, and another activation layer to obtain the second input feature map. The second input feature map is then input into an attention layer to obtain the target attention feature map. The target attention feature map and the first input feature map are then added together by residual processing to obtain the output feature map.

[0069] Example 5:

[0070] Figure 5 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the device includes:

[0071] The acquisition module 501 is used to acquire the target image to be processed;

[0072] Processing module 502 is used to perform convolution processing on the input target image based on the convolutional layers of a pre-trained hybrid attention super-resolution network model to obtain a shallow feature map; based on a predetermined number of cascaded attention residual layers in the hybrid attention super-resolution network model, the shallow feature map is sequentially passed through each attention residual layer for perceptual processing to obtain a deep feature map output by the last attention residual layer; based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of the corresponding rows and columns of the shallow feature map and the deep feature map are added to obtain the processed target feature map; based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain an output high-resolution image with enhanced details.

[0073] Further, the processing module 502 is specifically configured to, for each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, input the shallow feature map as the first input feature map into the attention residual layer; if the attention residual layer is not the first attention residual layer, use the output feature map of the previous attention residual layer as the first input feature map of the attention residual layer; perform perceptual processing on the first input feature map based on the attention layer of the attention residual layer to obtain the target attention feature map output by the attention layer after perceptual processing; and add the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map based on the residual addition processing layer of the attention residual layer to obtain the output feature map of the attention residual layer, until the deep feature map output by the last attention residual layer is obtained.

[0074] Further, the processing module 502 is specifically configured to: a processing unit of the attention layer based on the attention residual layer; input the pixel value of each pixel in the first input feature map and the pre-stored Local Binary Pattern (LBP) sampling function in the attention layer to obtain an output LBP feature value matrix and store it in the attention layer; input the first input feature map into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map; process the LBP feature value matrix through the second perceptron unit of the attention layer to obtain a gradient-based attention feature map; input the two attention feature maps into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map; and, based on the dot product processing unit of the attention layer, perform a dot product processing on the fused attention feature map and the first input feature map to obtain an output target attention feature map.

[0075] Furthermore, each attention residual layer also includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. The processing module 502 is specifically used to input the first input feature map into the first convolutional layer of the attention residual layer for convolution processing, through the first activation layer for activation processing, through the second convolutional layer for convolution processing, through the second activation layer for activation processing to obtain a second input feature map, and input the second input feature map into the attention layer of the attention residual layer for subsequent processing.

[0076] Example 6:

[0077] Figure 6 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Based on the above embodiments, this application also provides an electronic device, such as... Figure 6 As shown, it includes: processor 601, communication interface 602, memory 603 and communication bus 604, wherein processor 601, communication interface 602 and memory 603 communicate with each other through communication bus 604.

[0078] The memory 603 stores a computer program, which, when executed by the processor 601, causes the processor 601 to perform the following steps:

[0079] The target image to be processed is obtained, and a shallow feature map is obtained by performing convolution processing on the input target image based on the convolutional layer of the pre-trained hybrid attention super-resolution network model.

[0080] Based on a predetermined number of cascaded attention residual layers in the hybrid attention super-resolution network model, the shallow feature map is sequentially processed through each attention residual layer to obtain the deep feature map output by the last attention residual layer. Based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of the corresponding rows and columns of the shallow feature map and the deep feature map are added to obtain the processed target feature map. Based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain the output high-resolution image with enhanced details.

[0081] Furthermore, the deep feature map output by the last attention residual layer, obtained by sequentially processing the shallow feature map through each attention residual layer in the hybrid attention super-resolution network model with a predetermined number of cascaded attention residual layers, includes:

[0082] For each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map is used as the first input feature map and input into the attention residual layer. If the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is used as the first input feature map of the attention residual layer. The attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing. Based on the residual addition processing layer of the attention residual layer, the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map are added to obtain the output feature map of the attention residual layer, until the deep feature map output by the last attention residual layer is obtained.

[0083] Furthermore, the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing, including:

[0084] The processing unit of the attention layer based on the attention residual layer inputs the pixel value of each pixel in the first input feature map and the local binary pattern LBP sampling function pre-stored in the attention layer to obtain the output LBP feature value matrix and stores it in the attention layer.

[0085] The first input feature map is input into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map. The LBP feature value matrix is ​​then processed by the second perceptron unit of the attention layer to obtain a gradient-based attention feature map. The two attention feature maps are then input into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map. Finally, the fused attention feature map is multiplied by the first input feature map by the dot product processing unit of the attention layer to obtain the output target attention feature map.

[0086] Furthermore, each attention residual layer further includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. Before the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the perceptually processed attention layer, the method further includes:

[0087] The first input feature map is input into the first convolutional layer of the attention residual layer for convolution processing, then through the first activation layer for activation processing, then through the second convolutional layer for convolution processing, and then through the second activation layer for activation processing to obtain the second input feature map. The second input feature map is then input into the attention layer of the attention residual layer for subsequent processing.

[0088] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0089] The communication interface 602 is used for communication between the above-mentioned electronic device and other devices.

[0090] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0091] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0092] Example 7:

[0093] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0094] The target image to be processed is obtained, and a shallow feature map is obtained by performing convolution processing on the input target image based on the convolutional layer of the pre-trained hybrid attention super-resolution network model.

[0095] Based on a predetermined number of cascaded attention residual layers in the hybrid attention super-resolution network model, the shallow feature map is sequentially processed through each attention residual layer to obtain the deep feature map output by the last attention residual layer. Based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of the corresponding rows and columns of the shallow feature map and the deep feature map are added to obtain the processed target feature map. Based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain the output high-resolution image with enhanced details.

[0096] Furthermore, the deep feature map output by the last attention residual layer, obtained by sequentially processing the shallow feature map through each attention residual layer in the hybrid attention super-resolution network model with a predetermined number of cascaded attention residual layers, includes:

[0097] For each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map is used as the first input feature map and input into the attention residual layer. If the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is used as the first input feature map of the attention residual layer. The attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing. Based on the residual addition processing layer of the attention residual layer, the pixel values ​​of the corresponding rows and columns of the target attention feature map and the first input feature map are added to obtain the output feature map of the attention residual layer, until the deep feature map output by the last attention residual layer is obtained.

[0098] Furthermore, the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing, including:

[0099] The processing unit of the attention layer based on the attention residual layer inputs the pixel value of each pixel in the first input feature map and the local binary pattern LBP sampling function pre-stored in the attention layer to obtain the output LBP feature value matrix and stores it in the attention layer.

[0100] The first input feature map is input into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map. The LBP feature value matrix is ​​then processed by the second perceptron unit of the attention layer to obtain a gradient-based attention feature map. The two attention feature maps are then input into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map. Finally, the fused attention feature map is multiplied by the first input feature map by the dot product processing unit of the attention layer to obtain the output target attention feature map.

[0101] Furthermore, each attention residual layer further includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. Before the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the perceptually processed attention layer, the method further includes:

[0102] The first input feature map is input into the first convolutional layer of the attention residual layer for convolution processing, then through the first activation layer for activation processing, then through the second convolutional layer for convolution processing, and then through the second activation layer for activation processing to obtain the second input feature map. The second input feature map is then input into the attention layer of the attention residual layer for subsequent processing.

[0103] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized by, The method includes: The target image to be processed is obtained, and a shallow feature map is obtained by performing convolution processing on the input target image based on the convolutional layer of the pre-trained hybrid attention super-resolution network model. For each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map is used as the first input feature map and input to the attention residual layer. If the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is used as the first input feature map of the attention residual layer. The attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the perceptually processed attention layer. Based on the residual addition processing layer of the attention residual layer, the target... The pixel values ​​of corresponding rows and columns of the attention feature map and the first input feature map are added together to obtain the output feature map of the attention residual layer. This process continues until the deep feature map output by the last attention residual layer is obtained. Based on the target residual addition processing layer of the hybrid attention super-resolution network model, the pixel values ​​of corresponding rows and columns of the shallow feature map and the deep feature map are added together to obtain the processed target feature map. Based on the upsampling layer in the hybrid attention super-resolution network model, the target feature map is input into the upsampling layer to obtain the output high-resolution image with enhanced details. The attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the attention layer after perceptual processing, including: The processing unit of the attention layer based on the attention residual layer inputs the pixel value of each pixel in the first input feature map and the local binary pattern LBP sampling function pre-stored in the attention layer to obtain the output LBP feature value matrix and stores it in the attention layer. The first input feature map is input into the first perceptron unit of the attention layer for perceptual processing to obtain a brightness-based attention feature map. The LBP feature value matrix is ​​then processed by the second perceptron unit of the attention layer to obtain a gradient-based attention feature map. The two attention feature maps are then input into the fusion layer unit of the attention layer for fusion processing to obtain a fused attention feature map. Finally, the fused attention feature map is multiplied by the first input feature map by the dot product processing unit of the attention layer to obtain the output target attention feature map.

2. The method of claim 1, wherein, Each attention residual layer further includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. Before the attention layer based on the attention residual layer performs perceptual processing on the first input feature map to obtain the target attention feature map output by the perceptually processed attention layer, the method further includes: The first input feature map is input into the first convolutional layer of the attention residual layer for convolution processing, then through the first activation layer for activation processing, then through the second convolutional layer for convolution processing, and then through the second activation layer for activation processing to obtain the second input feature map. The second input feature map is then input into the attention layer of the attention residual layer for subsequent processing.

3. An image processing apparatus characterized by comprising: The device includes: The acquisition module is used to acquire the target image to be processed; The processing module is used to perform convolutional processing on the input target image based on the convolutional layers of a pre-trained hybrid attention super-resolution network model to obtain a shallow feature map. For each attention residual layer in the hybrid attention super-resolution network model, if the attention residual layer is the first attention residual layer, the shallow feature map is used as the first input feature map to the attention residual layer; if the attention residual layer is not the first attention residual layer, the output feature map of the previous attention residual layer is used as the first input feature map of the attention residual layer. Based on the attention layer of the attention residual layer, perceptual processing is performed on the first input feature map to obtain the target attention output by the perceptually processed attention layer. The target attention feature map is processed by adding the pixel values ​​of corresponding rows and columns of the target attention feature map and the first input feature map using the residual summing processing layer of the attention residual layer. This process continues until the deep feature map output by the last attention residual layer is obtained. Then, the target residual summing processing layer of the hybrid attention super-resolution network model adds the pixel values ​​of corresponding rows and columns of the shallow feature map and the deep feature map to obtain the processed target feature map. Finally, the target feature map is input into the upsampling layer of the hybrid attention super-resolution network model to obtain a high-resolution image with enhanced details. The processing module is specifically configured to: ...

4. The apparatus according to claim 3, characterized in that, Each attention residual layer further includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer. The processing module is specifically used to input the first input feature map into the first convolutional layer of the attention residual layer for convolution processing, through the first activation layer for activation processing, through the second convolutional layer for convolution processing, through the second activation layer for activation processing to obtain a second input feature map, and input the second input feature map into the attention layer of the attention residual layer for subsequent processing.

5. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the image processing method according to any one of claims 1-2.

6. A computer-readable storage medium, characterized in that, It stores a computer program executable by a processor, which, when run on the processor, causes the processor to perform the steps of the image processing method according to any one of claims 1-2.

Images