Image sharpening methods

Abstract

This application provides an image sharpening method, comprising: obtaining an image to be processed; inputting the image to be processed into a pre-trained sharpening neural network to extract texture details and edge details from the image to be processed; and overlaying the texture details and edge details back onto the image to be processed to obtain a sharpened target image. This solution extracts texture details and edge details from the input image using a trained sharpening neural network and overlays them back onto the original input image to obtain a sharpened image, enabling more accurate detail differentiation and improving sharpening effects.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image sharpening method. Background Art

[0002] Sharpening is a common image processing technique that aims to compensate for image contours, enhance image edges and grayscale transitions, and make the image clearer. Image sharpening is used to highlight the edges and contours of objects in the image, or the characteristics of certain linear target elements.

[0003] Generally, image sharpening focuses on extracting image details, and generally processes the image from a spatial perspective.

[0004] However, currently, the sharpened image obtained after image sharpening still has problems such as details being difficult to distinguish, and the sharpening effect is poor. Summary of the Invention

[0005] The present application provides an image sharpening method and device for solving the problem of poor image sharpening effect in existing methods.

[0006] In a first aspect, an embodiment of the present application provides an image sharpening method, comprising:

[0007] Get the image to be processed;

[0008] Inputting the image to be processed into a pre-trained sharpening neural network to extract texture details and edge details of the image to be processed; wherein the trained sharpening neural network is trained based on first detail features and second detail features of a plurality of sample images in a training sample set, the first detail features of each image are extracted from the image using the pre-trained sharpening neural network, and the second detail features of each image are obtained by calibrating the real details of the image;

[0009] The texture details and the edge details are overlapped back onto the image to be processed to obtain a sharpened target image.

[0010] In one possible design of the first aspect, the sharpening neural network includes at least one encoder and at least two decoders;

[0011] Accordingly, the step of inputting the image to be processed into a pre-acquired trained sharpening neural network to extract texture details and edge details of the image to be processed includes:

[0012] Inputting the image to be processed into the encoder, and encoding the input image to obtain an encoding result;

[0013] Inputting the encoding result into a first decoder of the at least two decoders, decoding the encoding result, and obtaining texture details of at least two frequency bands;

[0014] The encoding result is input into a second decoder of the at least two decoders, and the encoding result is decoded to obtain edge details of at least two frequency bands.

[0015] In another possible design of the first aspect, before inputting the image to be processed into a pre-acquired trained sharpening neural network to obtain texture details and edge details of the image to be processed, the method further includes:

[0016] Acquire the training sample set for performing sharpening neural network training, wherein the training sample set includes first detail features and second detail features of a plurality of sample images;

[0017] For each sample image in the training sample set, obtaining texture details of at least two frequency bands and edge details of at least two frequency bands of the sample image from the first detail feature of the sample image;

[0018] Acquire the real texture details and the real edge details of the sample image from the second detail features of the sample image;

[0019] For the sample image, obtaining a first loss value between texture details of each frequency band and true texture details, and obtaining a second loss value between edge details of each frequency band and true edge details;

[0020] Calculating a total sharpening loss corresponding to the sample image according to a preset weighted value, the first loss value, and the second loss value;

[0021] According to the total sharpening loss corresponding to each sample image in the training sample set, the preset pre-trained sharpening neural network is trained to obtain the trained sharpening neural network.

[0022] In another possible design of the first aspect, the encoder includes: a shallow feature extraction network and a deep feature extraction network connected to the shallow feature extraction network, the shallow feature extraction network including at least one processing module, and the deep feature extraction network including at least one hybrid attention group module;

[0023] Accordingly, encoding the input image to obtain an encoding result includes:

[0024] Extracting multi-scale features of the input image through convolution branches of different scales in the processing module and fusing them to obtain fused features;

[0025] Performing a coordinate attention mechanism calculation on the fused features through the coordinate attention mechanism module in the processing module to obtain a feature map weight, and multiplying the feature map weight by the fused features as the output of the shallow feature extraction network;

[0026] The hybrid attention group module extracts high-level image features from the output of the shallow feature extraction module, and the high-level image features are fused with the output of the shallow feature extraction module after convolution to obtain deep coding features as the coding result.

[0027] In another possible design of the first aspect, the shallow feature extraction network includes a first processing module, a second processing module, and a third processing module;

[0028] Accordingly, the output of the shallow feature extraction network is obtained, including:

[0029] The input image is processed by the first processing module to obtain a first shallow feature map of A*B*N, where A and B represent image sizes and N represents the number of image channels;

[0030] Processing the first shallow feature map by the second processing module to obtain a second shallow feature map, where the size of the second shallow feature map is smaller than that of the first shallow feature map and the number of channels is greater than that of the first shallow feature map;

[0031] The second shallow feature map is processed by the third processing module to obtain a third shallow feature map as the output of the shallow feature extraction network. The size of the third shallow feature map is smaller than that of the second shallow feature map and the number of channels is greater than that of the second shallow feature map.

[0032] In another possible design of the first aspect, the number of convolution kernels and the step size of the multi-scale branch modules in the first processing module, the second processing module, and the third processing module are different, and the number of convolution kernels and the step size of the coordinate attention mechanism modules in the first processing module, the second processing module, and the third processing module are different.

[0033] In another possible design of the first aspect, the mixed attention group module includes a mixed attention block;

[0034] Accordingly, extracting high-level image features from the output of the shallow feature extraction module through the hybrid attention group module includes:

[0035] Normalizing the output of the shallow feature extraction module by the hybrid attention block, processing it simultaneously by the channel attention mechanism and the window-based multi-head self-attention mechanism, and fusing it with the output of the shallow feature extraction module to obtain a first feature;

[0036] The first feature is subjected to layer normalization and a multi-layer perceptron by the hybrid attention block and then fused with the first feature to obtain the output of the hybrid attention block as the high-level image feature.

[0037] In another possible design of the first aspect, the first decoder includes a fourth processing module, a fifth processing module, and a sixth processing module;

[0038] Accordingly, obtaining texture details of at least two frequency bands includes:

[0039] Processing the encoding result by the fourth processing module to obtain a first texture detail feature map of A / 2*B / 2*2N;

[0040] Processing the first texture detail feature map by the fifth processing module to obtain a second texture detail feature map, wherein the size of the second texture detail feature map is larger than that of the first texture detail feature map and the number of channels of the second texture detail feature map is smaller than that of the first texture detail feature map;

[0041] The second texture detail feature map is processed by the sixth processing module to obtain a third texture detail feature map as the texture detail. The size of the third texture detail feature map is the same as the size of the second texture detail feature map, and the number of channels of the third texture detail feature map is the same as the number of frequency bands of the texture detail.

[0042] In another possible design of the first aspect, the second decoder includes a seventh processing module, an eighth processing module, and a ninth processing module;

[0043] Accordingly, obtaining edge details of at least two frequency bands includes:

[0044] The encoding result is processed by the seventh processing module to obtain a first edge detail feature map of A / 2*B / 2*2N;

[0045] Processing the first edge detail feature map by the eighth processing module to obtain a second edge detail feature map, wherein the size of the second edge detail feature map is larger than that of the first edge detail feature map and the number of channels of the second edge detail feature map is smaller than that of the first edge detail feature map;

[0046] The second edge detail feature map is processed by the ninth processing module to obtain a third edge detail feature map as the edge detail. The size of the third edge detail feature map is the same as the size of the second edge detail feature map, and the number of channels of the third edge detail feature map is the same as the number of frequency bands of the edge detail.

[0047] In a second aspect, an embodiment of the present application provides an image sharpening device, comprising:

[0048] An image acquisition module, used for acquiring images to be processed;

[0049] a detail extraction module, configured to input the image to be processed into a pre-trained sharpening neural network to extract texture details and edge details of the image to be processed; wherein the trained sharpening neural network is trained based on first detail features and second detail features of a plurality of sample images in a training sample set, the first detail features of each image are extracted from the image using the pre-trained sharpening neural network, and the second detail features of each image are obtained by calibrating the real details of the image;

[0050] The sharpened image acquisition module is used to overlay the texture details and the edge details onto the image to be processed to obtain a sharpened target image.

[0051] The image sharpening method, apparatus, device and readable storage medium provided in the embodiments of the present application obtain a trained sharpening neural network, extract the texture details and edge details of the input image based on the trained sharpening neural network, and overlay them back to the original input image to obtain a sharpened image, which can more accurately distinguish details and improve the sharpening effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 A schematic diagram of the overall process of image sharpening provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the process of an image sharpening method provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the overall network structure of the sharpened neural network provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the specific network structure of the sharpening neural network provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of the processing module provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the structure of the hybrid attention group module provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the structure of an image sharpening device provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] Sharpening is a common image processing technique that plays a key role in fields such as image signal processing (ISP). Its goal is to enhance image detail and improve image clarity, while maintaining minimal information change outside of the detail region. Image sharpening focuses on extracting image detail, typically by processing the image from a spatial perspective. Commonly used classical sharpening techniques utilize filtering methods such as unsharp masking, the Laplacian operator, and Difference of Gaussian to extract image detail and then overlay it onto the original image to enhance the original detail. However, these classical methods apply the same processing to every pixel, resulting in poor overall sharpening results. In recent years, a growing number of researchers have addressed the challenges of classical sharpening, including sharpening images in frequency bands or regions. However, these improved methods still suffer from numerous disadvantages, including low efficiency, inability to accurately distinguish texture and edge details, noise sensitivity, and poor scene adaptability. Currently, there is no research on the application of deep learning to image sharpening.

[0063] In response to the current problems, the present application provides an image sharpening method, apparatus, device and readable storage medium. By constructing and training a sharpening neural network, the texture details and edge details of the input image are extracted based on the trained sharpening neural network, and then overlaid back onto the original input image to obtain a sharpened image, which can more accurately distinguish details and improve the sharpening effect.

[0064] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0065] Figure 1 The overall process diagram of image sharpening provided in the embodiment of the present application is as follows: Figure 1 As shown in the figure, taking the image to be sharpened as a single-channel luminance image as an example, the multi-band texture details and multi-band edge details are extracted through the sharpening neural network, and then the i-th texture detail is selected from the multi-band texture details, and the j-th texture detail is selected from the multi-band edge details (i, j are constants), and the i-th texture detail band and the j-th edge detail band are selected to overlap with the image to be sharpened to obtain the final sharpening result.

[0066] Figure 2 This is a flow chart of an image sharpening method provided in an embodiment of the present application. The method can be applied to image processing devices such as Figure 2 As shown, it specifically includes the following steps: Step S201, obtaining an image to be processed.

[0067] In this embodiment, the image to be processed is the image to be sharpened. For example, the image to be processed can be a single-channel brightness image. The image usually includes certain elements, such as people, animals, plants, buildings, etc. These elements usually also have edge details and texture details.

[0068] In step S202, the image to be processed is input into a pre-trained sharpening neural network to extract texture details and edge details of the image to be processed. The trained sharpening neural network is trained based on first detail features and second detail features of multiple sample images in a training sample set. The first detail features of each image are extracted from the image using the pre-trained sharpening neural network, and the second detail features of each image are obtained by calibrating the actual details of the image.

[0069] In this embodiment, when the image to be processed is sharpened, it is necessary to input a pre-trained sharpening neural network (i.e., a trained sharpening neural network) to extract texture details and edge details. To this end, it is necessary to obtain a training sample set (including multiple sample images) before image sharpening, and construct a sharpening neural network (i.e., construct a pre-trained sharpening neural network) and a loss function for training. By inputting the sample image into the constructed sharpening neural network, a first detail feature is extracted (the first detail feature is extracted from the sample image by the pre-trained sharpening neural network and does not match the actual detail feature of the sample image), and then compared with the second detail feature of the sample image (i.e., the actual real detail of the sample image, which can be given in advance). By comparing the difference between the first detail feature and the second detail feature, as feedback, the parameters of the untrained sharpening neural network are adjusted and optimized until the difference between the first detail feature and the second detail feature is as small as possible, that is, the training is completed, and the trained sharpening neural network is obtained.

[0070] In this embodiment, an image edge refers to a local discontinuous feature within the image, typically determined by grayscale values. An image edge is a location where grayscale values ​​suddenly change, i.e., where the gradient is large. Image texture refers to recurring local patterns and their arrangement within the image. It can also be considered a repetitive pattern of localized changes in image intensity. Texture is generally used to describe the appearance of an object's surface. This information includes not only color but also various other aspects, such as normals, roughness, and ambient light.

[0071] Exemplarily, in some embodiments, a trained sharpening neural network can be obtained by the following steps: obtaining a training sample set for training the sharpening neural network, the training sample set including first detail features and second detail features of multiple sample images; for each sample image in the training sample set, obtaining texture details of at least two frequency bands and edge details of at least two frequency bands of the sample image from the first detail features of the sample image; obtaining real texture details and real edge details of the sample image from the second detail features of the sample image; for the sample image, obtaining a first loss value between the texture details of each frequency band and the real texture details, and obtaining a second loss value between the edge details of each frequency band and the real edge details; calculating the total sharpening loss corresponding to the sample image according to the preset weighted value, the first loss value and the second loss value; training the preset pre-trained sharpening neural network according to the total sharpening loss corresponding to each sample image in the training sample set to obtain a trained sharpening neural network.

[0072] In this embodiment, the loss function of the sharpening neural network can be expressed as:

[0073] L detail =Ltex +λL edge

[0074]

[0075]

[0076] In the above formula, Ldetail represents the total loss function of sharpening, L tex Represents the first loss value, L edge represents the second loss value, represents the i-th texture detail extracted by the sharpening neural network, represents the i-th real texture detail, represents the i-th edge detail extracted by the sharpening neural network, represents the i-th true edge detail, and λ is the weight of the edge detail loss function.

[0077] In this embodiment, by constructing a total sharpening loss function composed of texture and edge detail loss functions, and training with an Adam optimizer until the total sharpening loss function converges, the sharpening neural network can achieve better performance and further improve the accuracy of texture detail and edge detail extraction.

[0078] Step S203 , overlaying the texture details and edge details back onto the image to be processed to obtain a sharpened target image.

[0079] In this embodiment, the image to be processed is represented as Y, the sharpened image is represented as S(Y), and the texture details are represented as f tex , edge details are represented as f edge For example, back-stack can refer to the image to be processed Y and the texture details f tex , edge detailsf edge Add and sum to get S(Y). For example, the texture details f tex and edge details f edge Configure the corresponding sharpening strength coefficient, for example, for texture detail f tex Configure the sharpening strength S tex , is the edge detail f edge Configure the sharpening strength S edge , we can use the following formula to realize the overlap and calculate the sharpened image S(Y):

[0080] S(Y)=Y+f tex *S tex +fedge*S edge

[0081] The embodiment of the present application uses a neural network to simultaneously extract two different types of details, texture and edge, from an input image, and overlays them back onto the original image to achieve a sharpening effect. This can solve the problems of noise influence and sharpening blur that occur in the current image sharpening process, accurately distinguish between texture and edge details in the image, and improve the sharpening effect.

[0082] Figure 3 The overall network structure diagram of the sharpened neural network provided in the embodiment of the present application is as follows: Figure 3 As shown, the sharpening neural network includes at least one encoder and at least two decoders (one of which is a texture detail decoder and the other is an edge detail decoder); when the texture details and edge details of the image to be processed are extracted by the trained sharpening neural network, it can be specifically implemented through the following steps: inputting the image to be processed into the encoder, encoding the input image to obtain an encoding result; inputting the encoding result into the first decoder of at least two decoders, decoding the encoding result, and obtaining texture details of at least two frequency bands; inputting the encoding result into the second decoder of at least two decoders, decoding the encoding result, and obtaining edge details of at least two frequency bands.

[0083] In this embodiment, the sharpening neural network as a whole follows an encoder-decoder network architecture. The image input to the encoder can be the image to be processed as described above. An encoder is constructed to extract feature encoding of the input image Y, and then edge and texture detail decoders are used to decode and reconstruct the edge and texture details of multiple frequency bands of the input image Y. By constructing a texture detail decoder to extract texture details in the image, and constructing an edge detail decoder to extract edge details in the image, it is possible to accurately distinguish between edges and textures, further improving the sharpening effect of the image.

[0084] Among them, texture details and edge details can be divided into multiple frequency bands. For example, the texture details f tex Specifically, it can be expressed as: tex =[f tex1 , f tex2 ,...,f texi , f texNtex ], edge detail f edge Specifically, it can be expressed as: edge =[f edge1 , f edge2 ,...,f edgei , f edgeNedge ], that is, the texture details can be divided into Ntex frequency bands of texture details, and the edge details can be divided into Nedge segments of edge details. For example, f texNtex It can refer to high-frequency texture details, f tex1 Can refer to low-frequency texture details. edgeNedgeIt can refer to the edge details of the high-frequency band, f edge1 This can refer to low-frequency edge details.

[0085] Furthermore, in some other embodiments, Figure 4 The specific network structure diagram of the sharpening neural network provided in the embodiment of this application is as follows: Figure 4 As shown, the encoder may include a shallow feature extraction network and a deep feature extraction network connected to the shallow feature extraction network.

[0086] Among them, the shallow feature extraction network includes at least one processing module, and the deep feature extraction network includes at least one hybrid attention group module. The processing module can be a multi-scale branch block (MSB-CA) combined with the coordinate attention mechanism. The multi-scale branch block MSB is composed of four convolution branches of different scales. The outputs of multiple branches are added and then passed through an activation function (such as the Rectified Linear Unit (ReLU) function), that is, the multi-branch block that constitutes the whole replaces the traditional conv+relu mode to enrich the feature space. Then, the coordinate attention mechanism block CA is used to make the encoding process focus more on the detailed areas with high-frequency information in the image, while ignoring the flat areas without detailed information.

[0087] For example, Figure 5 A schematic diagram of the structure of the processing module provided in the embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, the multi-scale branch block MSB is composed of four convolution branches of different scales (specifically including 3x3 convolution + average pooling, 1x1 convolution + 3x3 convolution, 1x1 convolution + average pooling and 5x5 convolution), and the coordinate attention mechanism block CA includes horizontal average pooling, vertical average pooling, 1x1 convolution, activation function (such as ReLU activation function), and two 1x1 convolution branches in sequence.

[0088] In this embodiment, the deep feature extraction network includes at least one hybrid attention group module RHAG. When the deep feature extraction network includes multiple hybrid attention group modules RHAG modules, the respective hybrid attention group modules RHAG can be cascaded to extract more abstract high-level image features in the image.

[0089] In this embodiment, the input image can be encoded to obtain an encoding result through the following steps: multi-scale features of the input image are extracted and fused through convolution branches of different scales in the processing module to obtain fused features; the coordinate attention mechanism module in the processing module is used to calculate the coordinate attention mechanism on the fused features to obtain feature map weights, and the feature map weights are multiplied by the fused features to serve as the output of the shallow feature extraction network; high-level image features are extracted from the output of the shallow feature extraction module through the mixed attention group module, and the high-level image features are fused with the output of the shallow feature extraction module after convolution to obtain deep coding features as the encoding result.

[0090] In this embodiment, f in Denotes the input of the multi-scale branch block combined with the coordinate attention mechanism (for example, the input image Y), and uses f MCB-CA represents the output feature of the multi-scale branch block (i.e., processing module) combined with the coordinate attention mechanism, then the output of the multi-scale branch block MSB is MSB(f in ), f MCB-CA =CA(MSB(f in )).

[0091] For details, please refer to the above Figure 5 , first input f in (For example, image Y) passes through a multi-scale branch module consisting of four branches: 3x3 convolution + average pooling, 1x1 convolution + 3x3 convolution, 1x1 convolution + average pooling, and 5x5 convolution. The rich multi-scale features in the image are extracted and fused, and then Relu is used as the activation function to obtain the output of the multi-scale branch block MSB as f MSBout =MSB(f in ).in,

[0092] MSB(f in )=ReLU(AvgPool(conv 3X3 (f in ))+Conv 3x3 (Conv 1x1 (f in ))

[0093] +AvgPool(Conv 1x1 (f in ))+Conv 5x5 (f in ))

[0094] In this embodiment, the coordinate attention mechanism block CA implements the calculation of the coordinate attention mechanism on the output feature map of MSB, that is,

[0095] fMCB-CA =CA(f MSBout )

[0096] Specifically, first, the output f of the multi-scale branch block MSB is MSBout Complete the horizontal average pooling and the vertical average pooling, then concat splice and pass through the 1x1 convolution kernel and relu activation to get f cat ,

[0097] f cat =Relu(Conv 1x1 (Concat(AvgPool x (f MSBout ), AvgPooly(f MSBout ))))

[0098] In the above formula, AvgPool x () represents average pooling in the horizontal direction, and AvgPooly() represents average pooling in the vertical direction.

[0099] Among them, in getting f cat Afterwards, it is re-separated into horizontal and vertical features f catx and f caty Finally, the feature map weight is obtained through 1x1 and activation function (such as sigmoid function), and the output f of the multi-scale branch block MSB is obtained. MSBout The final output of the MSB-CA module (i.e., the processing module) is obtained by multiplication:

[0100] f MCB-CA =f MSBout *Sigmod(Conv 1x1 (f catx ))*sigmod(Conv 1x1 (f caty ))

[0101] In this embodiment, during the deployment and testing phase of the sharpening neural network, the branches of the multi-scale branch block MSB can be merged into a single-layer 5x5 convolution operation, which retains the advantages of the multi-scale convolution branches without increasing the model time consumption.

[0102] The embodiment of the present application adopts the MSB-CA module to form a shallow feature extraction network, introduces a multi-scale branch structure into the sharpening neural network, and adds a coordinate attention mechanism to make the sharpening neural network pay more attention to the high-frequency detail areas of the image, which can better extract image details. At the same time, the deep feature extraction network is composed of a transformer-based residual hybrid attention group RHAG module, which can extract more advanced and abstract features and further improve the sharpening effect.

[0103] Based on the above embodiments, in other embodiments, the shallow feature extraction network may include multiple of the above-mentioned processing modules (i.e., the multi-scale branch block MSB-CA combined with the coordinate attention mechanism). For example, taking the shallow feature extraction network as an example, which includes three processing modules connected in sequence, the output of the shallow feature extraction network can be obtained by the following steps: the input image is processed by the first processing module to obtain a first shallow feature map of A*B*N, where A and B represent the image size, and N represents the number of image channels; the first shallow feature map is processed by the second processing module to obtain a second shallow feature map, and the size of the second shallow feature map is smaller than the first shallow feature map and the number of channels is greater than the first shallow feature map; the second shallow feature map is processed by the third processing module to obtain a third shallow feature map as the output of the shallow feature extraction network, and the size of the third shallow feature map is smaller than the second shallow feature map and the number of channels is greater than the second shallow feature map.

[0104] In this embodiment, the input image is a single-channel brightness image Y with a pixel size of A*B. After being processed by the first processing module, the first shallow feature map f with a dimension of A*B*64 is obtained. s1 ,

[0105] f s1 =CA1(MSB1(Y))

[0106] In the above formula, CA1() represents the coordinate attention mechanism block CA in the first processing module, and MSB1() represents the multi-scale branch block in the first processing module.

[0107] Then f s1 After processing with the second processing module, the second shallow feature map f of A / 2*B / 2*128 is obtained, which has half the size but doubled the number of channels. s2 ,

[0108] f s2 =CA2(MSB2(f s1 ))

[0109] In the above formula, CA2() represents the coordinate attention mechanism block CA in the second processing module, and MSB2() represents the multi-scale branch block in the second processing module.

[0110] Finally, f s2 After being processed by the third processing module, the final third shallow feature map f is obtained, which is A / 4*B / 4*256 with the size halved and the number of feature channels doubled. s , further reducing the computational effort of the deep feature extraction module:

[0111] f s =CA3(MSB3(f s2 ))

[0112] In the above formula, CA3() represents the coordinate attention mechanism block CA in the third processing module, and MSB3() represents the multi-scale branch block in the third processing module.

[0113] Furthermore, in other embodiments, the number of convolution kernels and the step size of the multi-scale branch modules in the first processing module, the second processing module, and the third processing module are different, and the number of convolution kernels and the step size of the coordinate attention mechanism modules in the first processing module, the second processing module, and the third processing module are different.

[0114] For example, in order to ensure more accurate and efficient extraction of image details, the number of all convolution kernels in the multi-scale branch block MSB and the coordinate attention mechanism block CA in the first processing module can be configured to 64 with a step size of 1, the number of all convolution kernels in the multi-scale branch block MSB in the second processing module can be configured to 128 with a step size of 2, the number of convolution kernels of the coordinate attention mechanism block CA in the second processing module can be configured to 128 with a step size of 1, the number of convolution kernels of the multi-scale branch block MSB in the third processing module can be configured to 256 with a step size of 2, the number of convolution kernels of the coordinate attention mechanism block CA in the third processing module can be configured to 256 with a step size of 1.

[0115] For example, in other embodiments, Figure 6 A schematic diagram of the structure of the hybrid attention group module provided in the embodiment of the present application is shown as follows: Figure 6 As shown, the hybrid attention group module includes multiple hybrid attention blocks, and the output of the shallow feature extraction network is used as the input of the first hybrid attention block. After the high-level image features are extracted by the multiple hybrid attention blocks, the high-level image features are convolved and fused with the output of the shallow feature extraction module to obtain deep coding features as the encoding result. In this embodiment, the extraction of high-level image features can be achieved through the following steps: the output of the shallow feature extraction module is layer-normalized by the hybrid attention block, and is processed simultaneously by the channel attention mechanism and the window-based multi-head self-attention mechanism, and then fused with the output of the shallow feature extraction module to obtain the first feature; the first feature is layer-normalized and multi-layer perceptron-based by the hybrid attention block and then fused with the first feature to obtain the output of the hybrid attention block as the high-level image feature.

[0116] First, in this embodiment, the deep feature extraction network can include three cascaded hybrid attention group modules RHAG, and the structure of each hybrid attention group module RHAG is consistent (refer to Figure 6 ), the third shallow feature map f sInput three cascaded hybrid attention group modules RHAG to extract more abstract high-level features of the image. The third shallow feature map fs and the last hybrid attention group module RHAG undergo a layer of 3x3 convolution (the 3x3 convolution operation here introduces the inductive bias of the convolution operation into the network, which is more conducive to f s and the output f of the deep feature extraction network d The output of the deep feature extraction network is added and fused to obtain the output f d ,Right now

[0117] f d =f s +Conv 3x3 (f RHAG3 )

[0118] In the above formula, f RHAG3 represents the output of the third hybrid attention group module RHAG. RHAGi =RHAG i (f RHAGi-1 ), the value of i is 1, 2, 3, f RHAG0 =f s The fused f d It is still the feature map of A / 4*B / 4*256.

[0119] In this embodiment, continue to refer to Figure 6 Each hybrid attention group module RHAG consists of 6 hybrid attention blocks HAB and a layer of 3x3 convolution. The i-th hybrid attention group module RHAG outputs the feature f RHAGi It can be expressed as:

[0120] f RHAGi =f HABi，0 +Conv 3x3 (f HABi，6 )

[0121] f HABi，j =HAB i,j (f HABi，j-1 ), i takes the value of 1, 2, 3, indicating the i-th hybrid attention group module RHAG, and j takes the value of 1, 2, 3, 4, 5, 6, indicating the j-th hybrid attention block HAB.

[0122] In this embodiment, continue to refer to the above Figure 6 , each hybrid attention block HAB is used to: perform layer normalization on the feature X input to the hybrid attention block HAB, and then use channel attention and window-based multi-head self-attention mechanisms to process them simultaneously, and fuse them with feature X to obtain the first feature output X1:

[0123] X1=X+CAB(LN(X))+W-MSA(LN(X))*α

[0124] In the above formula, CAB is the channel attention mechanism, W-MSA is the window-based multi-head self-attention mechanism, LN is the layer normalization, and α is the fusion strength of the channel attention and multi-head self-attention mechanism.

[0125] Among them, based on the calculated first feature output X1, it is then fused with X1 through layer normalization and multi-layer perceptron to obtain the output HAB(X) of the hybrid attention block, which can be expressed as

[0126] HAB(X)=X1+MLP(LN(X1))

[0127] In the above formula, MLP is a multi-layer perceptron.

[0128] The embodiment of the present application constructs a sharpening neural network with a single encoder and dual decoder structure, which can simultaneously extract both texture and edge details in multiple frequency bands of the image. The MSB-CA module constitutes a shallow feature extraction network, introduces a multi-scale branch structure into the sharpening neural network, and adds a coordinate attention mechanism to enable the network to pay more attention to the high-frequency detail areas of the image, thereby better extracting image details; a deep feature extraction network is constructed using a transformer-based hybrid attention block HAB to extract more advanced and abstract features, further improving the sharpening effect of the image.

[0129] In some embodiments, the first decoder (ie, texture detail decoder) includes a fourth processing module, a fifth processing module, and a sixth processing module; wherein the structure of the fourth processing module is substantially the same as that of the fifth processing module, and the details can be referred to above. Figure 5 The sixth processing module includes Figure 5 In addition, the differences between the processing modules are that the number of convolution kernels and stride lengths of the multi-scale branch module are different, and the number of convolution kernels and stride lengths of the coordinate attention mechanism module are different.

[0130] In this embodiment, texture details of at least two frequency bands can be obtained through the following steps: the encoding result is processed by the fourth processing module to obtain a first texture detail feature map of A / 2*B / 2*2N; the first texture detail feature map is processed by the fifth processing module to obtain a second texture detail feature map, the size of the second texture detail feature map is larger than that of the first texture detail feature map and the number of channels of the second texture detail feature map is smaller than that of the first texture detail feature map; the second texture detail feature map is processed by the sixth processing module to obtain a third texture detail feature map as the texture detail, the size of the third texture detail feature map is the same as that of the second texture detail feature map and the number of channels of the third texture detail feature map is the same as the number of frequency bands of the texture detail.

[0131] In this embodiment, the encoding result can be regarded as the output f of the deep feature extraction network d , for the output f of the deep feature extraction network d , processed by the fourth processing module and then upsampled to obtain the first texture detail feature map of dimension A / 2*B / 2*128:

[0132] f tex1 =Upsample(CA tex1 (MSB tex1 (f d )))

[0133] In the above formula, f tex1 Represents the first texture detail feature map, Upsample() represents upsampling, CA tex1 Represents the coordinate attention mechanism block in the fourth processing module, MSB tex1 Represents the multi-scale branch block in the fourth processing module.

[0134] Then the first texture detail feature map f of A / 2*B / 2*128 tex1 After processing and upsampling by the fifth processing module, the second texture detail feature map with a size restored to A*B*64 is obtained:

[0135] f tex2 =Upsample(CA tex2 (MSB tex2 (f tex1 )))

[0136] In the above formula, f tex2 Represents the second texture detail feature map, Upsample() represents upsampling, CA tex2 Represents the coordinate attention mechanism block in the fifth processing module, MSB tex2 Represents the multi-scale branch block in the fifth processing module.

[0137] Finally, the second texture detail feature map of A*B*64 is processed by the sixth processing module to obtain a size of A*B*N tex The third texture detail feature map.

[0138] f tex =MSB tex3 (f tex2 )

[0139] In the above formula, f tex Represents the third texture detail feature map, MSB tex3 Represents the multi-scale branch block in the sixth processing module.

[0140] For example, in order to ensure more accurate and efficient extraction of texture details, the number of all convolution kernels of the multi-scale branch block MSB and the coordinate attention mechanism block CA in the fourth processing module can be configured to 128 with a step size of 1, the number of all convolution kernels of the multi-scale branch block MSB and the coordinate attention mechanism block CA in the fifth processing module can be configured to 64 with a step size of 1, and the number of convolution kernels of the multi-scale branch block MSB in the sixth processing module can all be configured to N tex , N tex Indicates the number of texture detail bands reconstructed by decoding.

[0141] Furthermore, in other embodiments, the second decoder (i.e., edge detail decoder) includes a seventh processing module, an eighth processing module, and a ninth processing module; wherein the structure of the second decoder is consistent with that of the first decoder, and the main difference is that the weight values ​​of the convolution kernels in each processing module are different. By configuring different weight values, the differentiated extraction of texture details and edge details can be achieved.

[0142] In this embodiment, edge details of at least two frequency bands can be obtained through the following steps: the encoding result is processed by the seventh processing module to obtain a first edge detail feature map of A / 2*B / 2*2N; the first edge detail feature map is processed by the eighth processing module to obtain a second edge detail feature map, the size of the second edge detail feature map is larger than that of the first edge detail feature map and the number of channels of the second edge detail feature map is smaller than that of the first edge detail feature map; the second edge detail feature map is processed by the ninth processing module to obtain a third edge detail feature map as the edge detail, the size of the third edge detail feature map is the same as that of the second edge detail feature map and the number of channels of the third edge detail feature map is the same as the number of frequency bands of the edge detail.

[0143] In this embodiment, the encoding result can be regarded as the output f of the deep feature extraction network. d , for the output f of the deep feature extraction network d, processed by the seventh processing module and then upsampled to obtain the first edge detail feature map of dimension A / 2*B / 2*128:

[0144] f edge1 =Upsample(CA edge1 (MSB edge1 (f d )))

[0145] In the above formula, f tex1 Represents the first edge detail feature map, Upsample() represents upsampling, CA edge1 Represents the coordinate attention mechanism block in the seventh processing module, MSB edge1 Represents the multi-scale branch block in the seventh processing module.

[0146] Then the first edge detail feature map f of A / 2*B / 2*128 edge1 After processing and upsampling by the eighth processing module, the second edge detail feature map with a size restored to A*B*64 is obtained:

[0147] F edge2 =Upsample(CA edge2 (MSB edge2 (f edge1 )))

[0148] In the above formula, f edge2 Represents the second edge detail feature map, Upsample() represents upsampling, CA edge2 Represents the coordinate attention mechanism block in the eighth processing module, MSB edge2 Represents the multi-scale branch block in the eighth processing module.

[0149] Finally, the second edge detail feature map of A*B*64 is processed by the ninth processing module to obtain a feature map of size A*B*N edge The third edge detail feature map.

[0150] f edge =MSB edge3 (f edge2 )

[0151] In the above formula, f edge Represents the third edge detail feature map, MSB edge3 Represents the multi-scale branch block in the ninth processing module.

[0152] For example, in order to ensure more accurate and efficient extraction of texture details, the number of all convolution kernels of the multi-scale branch block MSB and the coordinate attention mechanism block CA in the seventh processing module can be configured to 128 with a step size of 1, the number of all convolution kernels of the multi-scale branch block MSB and the coordinate attention mechanism block CA in the eighth processing module can be configured to 64 with a step size of 1, and the number of convolution kernels of the multi-scale branch block MSB in the ninth processing module can all be configured to N tedge , N edge Indicates the number of edge detail bands reconstructed by decoding.

[0153] The embodiment of the present application obtains image texture details and edge details in different frequency bands by using a simple and efficient decoder for decoding, wherein the number of frequency bands can be changed by adjusting the number of convolution kernels Ntex and Nedge of the decoder output layer, thereby improving the efficiency of image sharpening and reducing the complexity of the sharpening process.

[0154] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0155] Figure 7 This is a structural diagram of the image sharpening device provided in the embodiment of the present application, as shown in FIG. Figure 7 As shown, the image sharpening device 700 includes an image acquisition module 710, a detail extraction module 720, and a sharpened image acquisition module 730. The image acquisition module 710 is used to acquire the image to be processed. The detail extraction module 720 is used to input the image to be processed into a pre-acquired trained sharpening neural network to extract the texture details and edge details of the image to be processed; wherein the trained sharpening neural network is trained based on the first detail features and second detail features of multiple sample images in the training sample set, the first detail features of each image are extracted from the image using the pre-trained sharpening neural network, and the second detail features of each image are obtained by calibrating the real details of the image. The sharpened image acquisition module 730 is used to overlay the texture details and edge details back onto the image to be processed to obtain a sharpened target image.

[0156] Optionally, the sharpening neural network includes at least one encoder and at least two decoders; accordingly, the detail extraction module can be specifically used to: input the image to be processed into the encoder, encode the input image to obtain an encoding result; input the encoding result into the first decoder of at least two decoders, decode the encoding result, and obtain texture details of at least two frequency bands; input the encoding result into the second decoder of at least two decoders, decode the encoding result, and obtain edge details of at least two frequency bands.

[0157] Optionally, it also includes a neural network training module for obtaining a training sample set for sharpening neural network training, the training sample set including first detail features and second detail features of multiple sample images; for each sample image in the training sample set, obtaining texture details of at least two frequency bands and edge details of at least two frequency bands of the sample image from the first detail features of the sample image; obtaining real texture details and real edge details of the sample image from the second detail features of the sample image; for the sample image, obtaining a first loss value between the texture details of each frequency band and the real texture details, and obtaining a second loss value between the edge details of each frequency band and the real edge details; according to the preset weighted value, the first loss value and the second loss value, the total sharpening loss corresponding to the sample image is calculated; according to the total sharpening loss corresponding to each sample image in the training sample set, the preset pre-training sharpening neural network is trained to obtain a trained sharpening neural network.

[0158] Optionally, the encoder includes: a shallow feature extraction network and a deep feature extraction network connected to the shallow feature extraction network, the shallow feature extraction network includes at least one processing module, and the deep feature extraction network includes at least one hybrid attention group module;

[0159] Correspondingly, the detail extraction module can be specifically used to: extract multi-scale features of the input image through convolution branches of different scales in the processing module and fuse them to obtain fused features; calculate the coordinate attention mechanism on the fused features through the coordinate attention mechanism module in the processing module to obtain the feature map weights, and multiply the feature map weights with the fused features as the output of the shallow feature extraction network; extract high-level image features from the output of the shallow feature extraction module through the mixed attention group module, and fuse the high-level image features with the output of the shallow feature extraction module after convolution to obtain deep coding features as the coding result.

[0160] Optionally, the shallow feature extraction network includes a first processing module, a second processing module and a third processing module;

[0161] Correspondingly, the detail extraction module can be specifically used to: process the input image through the first processing module to obtain a first shallow feature map of A*B*N, where A and B represent the image size, and N represents the number of image channels; process the first shallow feature map through the second processing module to obtain a second shallow feature map, and the size of the second shallow feature map is smaller than the first shallow feature map and the number of channels is greater than the first shallow feature map; process the second shallow feature map through the third processing module to obtain a third shallow feature map as the output of the shallow feature extraction network, and the size of the third shallow feature map is smaller than the second shallow feature map and the number of channels is greater than the second shallow feature map.

[0162] Optionally, the number of convolution kernels and the step size of the multi-scale branch modules in the first processing module, the second processing module, and the third processing module are different, and the number of convolution kernels and the step size of the coordinate attention mechanism modules in the first processing module, the second processing module, and the third processing module are different.

[0163] Optionally, the hybrid attention group module includes a hybrid attention block;

[0164] Correspondingly, the detail extraction module can be specifically used to: perform layer normalization on the output of the shallow feature extraction module through the hybrid attention block, and after simultaneously processing it through the channel attention mechanism and the window-based multi-head self-attention mechanism, fuse it with the output of the shallow feature extraction module to obtain the first feature; perform layer normalization and multi-layer perceptron on the first feature through the hybrid attention block and fuse it with the first feature to obtain the output of the hybrid attention block as a high-level image feature.

[0165] Optionally, the first decoder includes a fourth processing module, a fifth processing module and a sixth processing module;

[0166] Correspondingly, the detail extraction module can be specifically used to: process the encoding result through the fourth processing module to obtain a first texture detail feature map of A / 2*B / 2*2N; process the first texture detail feature map through the fifth processing module to obtain a second texture detail feature map, the size of the second texture detail feature map is larger than the first texture detail feature map and the number of channels of the second texture detail feature map is smaller than the first texture detail feature map; process the second texture detail feature map through the sixth processing module to obtain a third texture detail feature map as the texture detail, the size of the third texture detail feature map is the same as the size of the second texture detail feature map and the number of channels of the third texture detail feature map is the same as the number of frequency bands of the texture detail.

[0167] Optionally, the second decoder includes a seventh processing module, an eighth processing module and a ninth processing module;

[0168] Correspondingly, the detail extraction module can be specifically used to: process the encoding result through the seventh processing module to obtain a first edge detail feature map of A / 2*B / 2*2N; process the first edge detail feature map through the eighth processing module to obtain a second edge detail feature map, the size of the second edge detail feature map is larger than the first edge detail feature map and the number of channels of the second edge detail feature map is smaller than the first edge detail feature map; process the second edge detail feature map through the ninth processing module to obtain a third edge detail feature map as the edge detail, the size of the third edge detail feature map is the same as the size of the second edge detail feature map and the number of channels of the third edge detail feature map is the same as the number of frequency bands of the edge detail.

[0169] The device provided in the embodiments of the present application can be used to execute the method in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0170] It should be noted that the division of the modules of the above devices is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules can be implemented entirely in software called by a processing element, or entirely in hardware. Alternatively, some modules can be implemented in software called by a processing element, while others can be implemented in hardware. For example, the image acquisition module can be a separate processing element, or it can be integrated into a component of the above devices. Furthermore, it can be stored in the memory of the above devices in the form of program code, which can be called by a processing element of the above devices to perform the functions of the above image acquisition module. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or by software instructions.

[0171] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 8 As shown, the electronic device 800 includes: at least one processor 801 , a memory 802 , a bus 803 and a communication interface 804 .

[0172] The processor 801, the communication interface 804, and the memory 802 communicate with each other via the bus 803. The communication interface 804 is used to communicate with other devices. This communication interface includes a communication interface for data transmission and a display interface or operation interface for human-computer interaction.

[0173] The processor 801 is used to execute a computer program stored in the memory and run on the processor, and specifically can execute the relevant steps of the method described in the above embodiment. The processor may be a central processing unit, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device can be processors of the same type, such as one or more CPUs; or different types of processors, such as one or more CPUs and one or more ASICs.

[0174] The memory is used to store computer programs that can be run on the processor. The memory may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk storage.

[0175] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When at least one processor of an electronic device executes the computer instructions, the electronic device executes the methods provided in the various embodiments described above.

[0176] This embodiment further provides a computer program product, comprising computer instructions stored in a readable storage medium. At least one processor of an electronic device can read the computer instructions from the readable storage medium, and at least one processor can execute the computer instructions to cause the electronic device to implement the methods provided in the various embodiments described above.

[0177] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0178] It is understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the order of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image sharpening method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a pre-trained sharpening neural network to extract texture details and edge details of the image to be processed; wherein the trained sharpening neural network is trained based on first detail features and second detail features of a plurality of sample images in a training sample set, the first detail features of each image are extracted from the image using the pre-trained sharpening neural network, and the second detail features of each image are obtained by calibrating the real details of the image; Overlaying the texture details and the edge details back onto the image to be processed to obtain a sharpened target image; The sharpening neural network includes at least one encoder and at least two decoders; Accordingly, the step of inputting the image to be processed into a pre-acquired trained sharpening neural network to extract texture details and edge details of the image to be processed includes: Inputting the image to be processed into the encoder, and encoding the input image to obtain an encoding result; Inputting the encoding result into a first decoder of the at least two decoders, decoding the encoding result, and obtaining texture details of at least two frequency bands; The encoding result is input into a second decoder of the at least two decoders, and the encoding result is decoded to obtain edge details of at least two frequency bands.

2. The method according to claim 1, characterized in that Before inputting the image to be processed into a pre-acquired trained sharpening neural network to obtain texture details and edge details of the image to be processed, the method further includes: Acquire the training sample set for performing sharpening neural network training, wherein the training sample set includes first detail features and second detail features of a plurality of sample images; For each sample image in the training sample set, obtaining texture details of at least two frequency bands and edge details of at least two frequency bands of the sample image from the first detail feature of the sample image; Acquire the real texture details and the real edge details of the sample image from the second detail features of the sample image; For the sample image, obtaining a first loss value between texture details of each frequency band and true texture details, and obtaining a second loss value between edge details of each frequency band and true edge details; Calculating a total sharpening loss corresponding to the sample image according to a preset weighted value, the first loss value, and the second loss value; According to the total sharpening loss corresponding to each sample image in the training sample set, the preset pre-trained sharpening neural network is trained to obtain the trained sharpening neural network.

3. The method according to claim 1, characterized in that The encoder comprises: a shallow feature extraction network and a deep feature extraction network connected to the shallow feature extraction network, the shallow feature extraction network comprising at least one processing module, and the deep feature extraction network comprising at least one hybrid attention group module; Accordingly, encoding the input image to obtain an encoding result includes: Extracting multi-scale features of the input image through convolution branches of different scales in the processing module and fusing them to obtain fused features; Performing a coordinate attention mechanism calculation on the fused features through the coordinate attention mechanism module in the processing module to obtain a feature map weight, and multiplying the feature map weight by the fused features as the output of the shallow feature extraction network; The hybrid attention group module extracts high-level image features from the output of the shallow feature extraction module, and the high-level image features are fused with the output of the shallow feature extraction module after convolution to obtain deep coding features as the coding result.

4. The method according to claim 3, characterized in that The shallow feature extraction network includes a first processing module, a second processing module and a third processing module; Accordingly, the output of the shallow feature extraction network is obtained, including: The input image is processed by the first processing module to obtain The first shallow feature map, A and B represent the image size, and N represents the number of image channels; Processing the first shallow feature map by the second processing module to obtain a second shallow feature map, where the size of the second shallow feature map is smaller than that of the first shallow feature map and the number of channels is greater than that of the first shallow feature map; The second shallow feature map is processed by the third processing module to obtain a third shallow feature map as the output of the shallow feature extraction network. The size of the third shallow feature map is smaller than that of the second shallow feature map and the number of channels is greater than that of the second shallow feature map.

5. The method according to claim 4, characterized in that The number of convolution kernels and the step size of the multi-scale branch modules in the first processing module, the second processing module, and the third processing module are different, and the number of convolution kernels and the step size of the coordinate attention mechanism modules in the first processing module, the second processing module, and the third processing module are different.

6. The method according to claim 3, characterized in that The hybrid attention group module includes a hybrid attention block; Accordingly, extracting high-level image features from the output of the shallow feature extraction module through the hybrid attention group module includes: Normalizing the output of the shallow feature extraction module by the hybrid attention block, processing it simultaneously by the channel attention mechanism and the window-based multi-head self-attention mechanism, and fusing it with the output of the shallow feature extraction module to obtain a first feature; The first feature is subjected to layer normalization and a multi-layer perceptron by the hybrid attention block and then fused with the first feature to obtain the output of the hybrid attention block as the high-level image feature.

7. The method according to claim 1, characterized in that The first decoder includes a fourth processing module, a fifth processing module and a sixth processing module; Accordingly, obtaining texture details of at least two frequency bands includes: The encoding result is processed by the fourth processing module to obtain A and B represent image sizes, and N represents the number of image channels; Processing the first texture detail feature map by the fifth processing module to obtain a second texture detail feature map, wherein the size of the second texture detail feature map is larger than that of the first texture detail feature map and the number of channels of the second texture detail feature map is smaller than that of the first texture detail feature map; The second texture detail feature map is processed by the sixth processing module to obtain a third texture detail feature map as the texture detail. The size of the third texture detail feature map is the same as the size of the second texture detail feature map, and the number of channels of the third texture detail feature map is the same as the number of frequency bands of the texture detail.

8. The method according to claim 1, characterized in that The second decoder includes a seventh processing module, an eighth processing module and a ninth processing module; Accordingly, obtaining edge details of at least two frequency bands includes: The seventh processing module processes the encoding result to obtain The first edge detail feature map; A and B represent the image size, and N represents the number of image channels; Processing the first edge detail feature map by the eighth processing module to obtain a second edge detail feature map, wherein the size of the second edge detail feature map is larger than that of the first edge detail feature map and the number of channels of the second edge detail feature map is smaller than that of the first edge detail feature map; The second edge detail feature map is processed by the ninth processing module to obtain a third edge detail feature map as the edge detail. The size of the third edge detail feature map is the same as the size of the second edge detail feature map, and the number of channels of the third edge detail feature map is the same as the number of frequency bands of the edge detail.

9. An image sharpening device, characterized in that: include: An image acquisition module, used for acquiring images to be processed; a detail extraction module, configured to input the image to be processed into a pre-trained sharpening neural network to extract texture details and edge details of the image to be processed; wherein the trained sharpening neural network is trained based on first detail features and second detail features of a plurality of sample images in a training sample set, the first detail features of each image are extracted from the image using the pre-trained sharpening neural network, and the second detail features of each image are obtained by calibrating the real details of the image; a sharpened image acquisition module, configured to overlay the texture details and the edge details onto the image to be processed to obtain a sharpened target image; The sharpening neural network includes at least one encoder and at least two decoders; Correspondingly, the detail extraction module is specifically used to: input the image to be processed into the encoder, encode the input image to obtain an encoding result; input the encoding result into the first decoder of the at least two decoders, decode the encoding result, and obtain texture details of at least two frequency bands; input the encoding result into the second decoder of the at least two decoders, decode the encoding result, and obtain edge details of at least two frequency bands.

Images