Image enhancement network acquisition method, image enhancement method and device

Abstract

The present application discloses an image enhancement network, an image enhancement method and a device, wherein the image enhancement network comprises: a self-attention module, N encoding modules, N feature sequence modules, N cross-attention modules and N decoding modules; the feature sequence module downsamples the feature map output by the image to be enhanced or the previous feature sequence module to obtain a sequence feature map; the cross-attention module fuses the sequence feature map with the feature map output by the previous encoding module based on the cross-attention mechanism to obtain a fused feature map; the encoding module extracts features from the image to be enhanced or the fused feature map to obtain a third feature map; the self-attention module enhances the features of the fused feature map output by the Nth cross-attention module to obtain an enhanced feature map; the decoding module fuses the enhanced feature map or the third feature map with the image output by the previous decoding module and decodes to obtain an enhanced image. The above scheme improves the effect of image enhancement.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image enhancement network acquisition method, an image enhancement method and a device. Background Art

[0002] In harsh weather conditions, the images captured by the on-board visual sensors of the intelligent driving system may have low image quality in part or all of the field of view, such as when the images are blocked by rain and snow, blocked by raindrops or dirt, or blurred due to excessive rain or fog.

[0003] At present, when performing image enhancement processing such as removing rain and snow, removing raindrops or dirt, and deblurring on these low-quality images, the accuracy of the image enhancement model is mainly improved by deepening the encoder and decoder. This solution has the problem of poor image enhancement effect due to the large amount of image information loss. Summary of the invention

[0004] The embodiments of the present application provide a method for acquiring an image enhancement network, an image enhancement method and a device, thereby at least to a certain extent solving the problem of poor image enhancement effect of the image enhancement network.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a method for obtaining an image enhancement network is provided, wherein the image enhancement network comprises: a self-attention module, N encoding modules, N feature sequence modules, N cross attention modules and N decoding modules, wherein N is an integer greater than 1; each feature sequence module is connected in series, the feature sequence module corresponds one-to-one to the cross attention module, the output end of the feature sequence module is also connected to the input end of the cross attention module, the input data of the first feature sequence module is an image to be enhanced, the encoding module and the cross attention module are arranged at intervals and connected in series, the input data of the first encoding module is the image to be enhanced, the Nth cross attention module is connected to the first decoding module via the self-attention module, the decoding modules are connected in series, the first N-1 encoding modules correspond one-to-one to the last N-1 decoding modules and are jump-connected; wherein,

[0007] The feature sequence combination module is used to downsample the image to be enhanced or the feature map output by the previous feature sequence combination module to obtain a sequence feature map;

[0008] The cross attention module is used to perform feature fusion on the combined sequence feature map and the feature map output by the previous encoding module based on the cross attention mechanism to obtain a fused feature map;

[0009] The encoding module is used to extract features from the image to be enhanced or the fused feature map to obtain a third feature map;

[0010] The self-attention module is used to enhance the features of the fused feature map output by the Nth cross-attention module to obtain an enhanced feature map;

[0011] The decoding module is used to fuse the enhanced feature map or the third feature map with the image output by the previous decoding module and then decode it to obtain an enhanced image.

[0012] In some embodiments of the present application, based on the above scheme, downsampling the image to be enhanced or the feature map output by the previous feature sequence combination module to obtain the sequence feature map includes:

[0013] Perform pixel traversal on the image to be enhanced or the feature map output by the previous feature sequence combination module;

[0014] Arrange multiple pixels that are integer multiples of the pixel step length obtained by distance traversal to obtain a down-sampled image corresponding to the pixel;

[0015] The downsampled images corresponding to all pixels in the feature map output by the previous feature sequence combination module or the image to be enhanced are arranged side by side to obtain the sequence combination feature map.

[0016] In some embodiments of the present application, based on the aforementioned solution, the feature fusion of the combined sequence feature map and the feature map output by the previous encoding module based on the cross attention mechanism to obtain a fused feature map includes:

[0017] Performing a linear transformation on the combined sequence feature graph to obtain a first multi-head tensor and a second multi-head tensor;

[0018] Performing a linear transformation on the feature map output by the previous encoding module to obtain a third multi-head tensor;

[0019] Based on the cross attention mechanism, each head of the first multi-head tensor, the second multi-head tensor and the third multi-head tensor is operated to obtain the fused feature map.

[0020] In some embodiments of the present application, based on the above scheme, the attention mechanism is:

[0021]

[0022] Among them, Q is the first multi-head tensor, K is the second multi-head tensor, V is the third multi-head tensor, and d is the dimension of the feature in the multi-head tensor.

[0023] In some embodiments of the present application, based on the aforementioned scheme, the number N is 4, the first encoding module and the second encoding module both adopt convolutional self-attention neural networks, and the third encoding module and the fourth encoding module both adopt multi-head self-attention neural networks.

[0024] In some embodiments of the present application, based on the aforementioned solution, the step of extracting features from the image to be enhanced or the fused feature map to obtain a third feature map includes:

[0025] Down-sampling the image to be enhanced or the fused feature map using a convolutional neural network to obtain a down-fused feature map;

[0026] The convolutional self-attention neural network or the multi-head self-attention neural network is used to perform feature extraction on the lower fusion feature map to obtain the third feature map.

[0027] In some embodiments of the present application, based on the above scheme, the feature enhancement of the fused feature map output by the Nth cross attention module includes:

[0028] Determining a learnable parameter according to the size of the fused feature map output by the Nth cross attention module;

[0029] The fused feature map output by the Nth cross-attention module is feature enhanced according to the learnable parameters.

[0030] In some embodiments of the present application, based on the aforementioned solution, the step of enhancing the features of the fused feature map output by the Nth cross attention module according to the learnable parameter includes:

[0031] The pixel of each spatial position point of the fusion feature map output by the Nth cross-attention module is added to the learnable parameter and then the feature is enhanced.

[0032] According to a second aspect of an embodiment of the present application, there is provided an image enhancement method, comprising:

[0033] Acquire an image to be enhanced;

[0034] The image to be enhanced is input into the above-mentioned image enhancement network to obtain an enhanced image output by the image enhancement network.

[0035] According to a third aspect of an embodiment of the present application, an image enhancement device is provided, comprising a processor and a memory, wherein the memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements instructions of the method described in any one of the second aspects above.

[0036] In the present application, a self-attention module, N encoding modules, N feature sequence modules, N cross-attention modules and N decoding modules are set in the image enhancement network; each feature sequence module is connected in series, the feature sequence module corresponds to the cross-attention module one by one, the output end of the feature sequence module is also connected to the input end of the cross-attention module, the input data of the first feature sequence module is the image to be enhanced, the encoding module and the cross-attention module are arranged at intervals and connected in series, the input data of the first encoding module is the image to be enhanced, the Nth cross-attention module is connected to the first decoding module via the self-attention module, the decoding modules are connected in series, and the first N-1 encoding modules are one-to-one with the last N-1 decoding modules. The feature combination module is used to downsample the feature map output by the image to be enhanced or the previous feature combination module to obtain a combined feature map; the cross-attention module is used to fuse the combined feature map with the feature map output by the previous encoding module based on the cross-attention mechanism to obtain a fused feature map; the encoding module is used to extract features from the image to be enhanced or the fused feature map to obtain a third feature map; the self-attention module is used to enhance the features of the fused feature map output by the Nth cross-attention module to obtain an enhanced feature map; the decoding module is used to fuse the enhanced feature map or the third feature map with the image output by the previous decoding module and then decode to obtain an enhanced image. More features of the image to be enhanced can be retained in the image encoding stage, which improves the accuracy of the image enhancement network and the effect of image enhancement.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0039] Figure 1 A schematic diagram of the structure of an image enhancement network in one embodiment is shown;

[0040] Figure 2 A schematic diagram of the structure of an image enhancement network in another embodiment is shown;

[0041] Figure 3 Shows Figure 2 The structural diagram of the feature sequence combination module;

[0042] Figure 4 A schematic diagram of a data flow of an encoding structure in one embodiment is shown;

[0043] Figure 5 A schematic diagram of a process flow of an image enhancement method in an embodiment is shown;

[0044] Figure 6 A block diagram of an image enhancement device in one embodiment is shown;

[0045] Figure 7 A schematic structural diagram of an image enhancement device in an embodiment is shown. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0048] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0049] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0050] It should also be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0051] Figure 1 FIG. 4 shows a schematic diagram of the structure of an image enhancement network in an embodiment. Figure 1 As shown, the image enhancement network may include a self-attention module, N encoding modules, N feature sequence modules, N cross attention modules and N decoding modules, where N is an integer greater than 1; each feature sequence module is connected in series, the feature sequence module corresponds to the cross attention module one by one, the output end of the feature sequence module is also connected to the input end of the cross attention module, the input data of the first feature sequence module is the image to be enhanced, the encoding module and the cross attention module are arranged at intervals and connected in series, the input data of the first encoding module is the image to be enhanced, the Nth cross attention module is connected to the first decoding module via the self-attention module, the decoding modules are connected in series, the first N-1 encoding modules are respectively connected to the last N-1 decoding modules The coding modules correspond to each other one by one and are jump-connected; wherein, the feature combination module is used to downsample the feature map output by the image to be enhanced or the previous feature combination module to obtain a combination feature map; the cross-attention module is used to fuse the combination feature map with the feature map output by the previous encoding module based on the cross-attention mechanism to obtain a fused feature map; the encoding module is used to extract features from the image to be enhanced or the fused feature map to obtain a third feature map; the self-attention module is used to enhance the features of the fused feature map output by the Nth cross-attention module to obtain an enhanced feature map; the decoding module is used to fuse the enhanced feature map or the third feature map with the image output by the previous decoding module and decode it to obtain an enhanced image.

[0052] It should be understood that the feature combination modules are connected in series. The first feature combination module is used to downsample the image to be enhanced, and the 2nd to Nth feature combination modules are used to downsample the feature map output by the previous feature combination module. Since the feature combination modules correspond one-to-one to the cross-attention modules, the output end of the feature combination module is also connected to the input end of the cross-attention module. Therefore, the combined feature map obtained by downsampling the first feature combination module will be input into the first cross-attention module, and the combined feature maps obtained by downsampling the 2nd to Nth feature combination modules will be input into the 2nd to N cross-attention modules respectively.

[0053] The encoding module and the cross-attention module are arranged at intervals and connected in series. It can be understood that the connection order of the encoding module and the cross-attention module is: the first encoding module, the first cross-attention module, the second encoding module, the second cross-attention module... the Nth encoding module, the Nth cross-attention module. The first encoding module is used to extract features from the image to be enhanced. The first cross-attention module is used to fuse the combined sequence feature map obtained by downsampling the first feature combined sequence module with the feature map extracted by the first encoding module based on the cross-attention mechanism to obtain a fused feature map; the second encoding module is used to extract features from the fused feature map output by the first cross-attention module; the second cross-attention module fuses the combined sequence feature map obtained by downsampling the second feature combined sequence module with the feature map extracted by the second encoding module based on the cross-attention mechanism; the third encoding module extracts features from the fused feature map output by the second cross-attention module... and so on, until the Nth cross-attention module completes feature fusion, the self-attention module performs feature enhancement on the fused feature map output by the Nth cross-attention module to obtain an enhanced feature map, and the encoding stage ends and enters the decoding stage.

[0054] The first decoding module decodes the enhanced feature map to obtain an enhanced image. The second decoding module fuses the image output by the first decoding module with the third feature map output by the N-1th encoding module and decodes it. The third decoding module fuses the image output by the second decoding module with the third feature map output by the N-2th encoding module and decodes it... and so on, until the Nth decoding module completes decoding and outputs the final enhanced image.

[0055] Figure 2 FIG. 4 shows a schematic diagram of the structure of an image enhancement network in another embodiment, in which the number N is 4. Figure 2 As shown in the figure, the image enhancement network includes: a self-attention module, 4 encoding modules, 4 feature combination modules, 4 cross-attention modules and 4 decoding modules. The image to be enhanced is input into the first encoding module and the first feature combination module respectively. After being processed by the image enhancement network, the enhanced image is obtained.

[0056] Specifically, the self-attention module, four encoding modules, four feature sequence modules, and four cross-attention modules can be divided into four-layer encoding structures, wherein the first-layer encoding structure includes the first encoding module, the second-layer encoding structure includes the first feature sequence module, the first cross-attention module, and the second encoding module, the third-layer encoding structure includes the second feature sequence module, the second cross-attention module, and the third encoding module, and the fourth-layer encoding structure includes the third feature sequence module, the third cross-attention module, the fourth feature sequence module, the fourth cross-attention module, the fourth encoding module, and the self-attention module.

[0057] The present embodiment proposes a coding structure with a multi-level jump mechanism based on a feature sequence module, a cross attention module and a coding module, wherein the feature sequence module can retain as much effective input information as possible, including high-frequency pixel-level features and low-frequency semantic features, while downsampling the image or feature map to be enhanced. The cross attention module can integrate this information into the coding module in the form of an attention feature map. With the support of these two modules, the coding structure in the present technical solution can retain the visual information of each frequency band of the image to be enhanced to the greatest extent, and pass this information to the decoding module for image enhancement. Compared with the existing image enhancement solutions, the image enhancement network structure proposed in the present technical solution can greatly optimize the model, so that it has higher accuracy and image enhancement effect under different image enhancement tasks, and because the module itself has the characteristics of small number of parameters and low computing power consumption, the image enhancement network has fewer parameters added than the traditional autoencoder structure, so it has less impact on the network parameter amount and running speed.

[0058] In one embodiment, downsampling the feature map output by the image to be enhanced or the previous feature sequence combination module to obtain the sequence feature map may include the following steps: performing pixel traversal on the feature map output by the image to be enhanced or the previous feature sequence combination module; arranging a plurality of pixels that are integer multiples of the pixel step length obtained by the distance traversal to obtain a downsampled image corresponding to the pixel; and arranging the downsampled images corresponding to all pixels in the feature map output by the image to be enhanced or the previous feature sequence combination module side by side to obtain the sequence feature map.

[0059] Figure 3 Shows Figure 2 The structural diagram of the feature sequence module is as follows: Figure 3 As shown, assuming that the size of the input image is (W, H, C 输入) And the downsampling step size is 2. Starting from the first pixel, pixels whose pixel distances in the left, right, up, and down directions from this pixel are integer multiples of the step size are grouped together. This group of pixels is arranged according to the arrangement method of the input image to form the downsampled image. The multiple obtained images are arranged side by side in the channel direction to finally form the output of the feature ordering module. The size of the input image is (W, H, C 输入 ), and the size of the image obtained after feature ordering is (W / 2, H / 2, C 输入 *4). This feature ordering module can downsample the original input image while尽可能 retaining the original input information, retaining both the semantic information in the original image and the high-frequency details in the image. The formed output can maximize the compensation for the high-frequency detail information lost due to the progressive transmission of the multi-level coding structure. Therefore, the image enhancement network can restore each affected pixel in the image to be enhanced in high definition.

[0060] Figure 4 shows a data flow schematic diagram of the coding structure in an embodiment. The coding structure includes a feature ordering module, a cross-attention module, and a coding module, which can be Figure 2 the first feature ordering module, the first cross-attention module, and the second coding module in Figure 2 ; or it can be the second feature ordering module, the second cross-attention module, and the third coding module in

[0061] In an embodiment, based on the cross-attention mechanism, the ordered feature map is fused with the feature map output by the previous coding module to obtain a fused feature map, including: performing a linear transformation on the ordered feature map to obtain a first multi-head tensor and a second multi-head tensor; performing a linear transformation on the feature map output by the previous coding module to obtain a third multi-head tensor; based on the cross-attention mechanism, performing operations on each head of the first multi-head tensor, the second multi-head tensor, and the third multi-head tensor to obtain the fused feature map.

[0062] In the i-th level coding structure (0 < i < 5), the feature map extracted by the previous level coding module can be defined as the feature map F input to the cross-attention module in this level coding structure i-1 , and the ordered feature map P output by the feature ordering module in this level i-1 . The ordered feature map P i-1 is respectively input into two linear layers to obtain two multi-head tensors with the same size and the same number of heads, and is defined as query (Q) and key (K). At the same time, the feature map F i-1Input into a linear layer to get a multi-head tensor with the same size and number of heads as the query (Q) and keyword (K), and define it as value (V). Perform self-attention operation on each head of these three tensors to get the fused feature map R output by the cross-attention module at this level. i-1 , the encoding module at this level is used to fusion feature map R i-1 Feature extraction is performed to obtain the third feature map that is finally output by the hierarchical encoding structure.

[0063] In an example, the attention mechanism can be defined as:

[0064]

[0065] Among them, Q is the first multi-head tensor, K is the second multi-head tensor, V is the third multi-head tensor, d is the dimension of the feature in the multi-head tensor, and softmax is the softmax activation function.

[0066] In one embodiment, the first encoding module and the second encoding module both use convolutional self-attention neural networks, and the third encoding module and the fourth encoding module both use multi-head self-attention neural networks.

[0067] Performing feature extraction on the image to be enhanced or the fused feature map to obtain a third feature map may include the following steps: downsampling the image to be enhanced or the fused feature map using a convolutional neural network to obtain a lower fused feature map; performing feature extraction on the lower fused feature map using a convolutional self-attention neural network or a multi-head self-attention neural network to obtain a third feature map.

[0068] Assume that the fusion feature map input to the i-th encoding module is R i-1 , the third feature map output by the i-th encoding module is F i , and let F0 = I, I is the image to be enhanced, then in this embodiment, F i It can be expressed by the following formula:

[0069]

[0070] Among them, D i-1 For the fusion feature map R i-1 The down-sampling fused feature map, Conv2d s＝2 is a convolutional neural network with a step size of 2, ConvAttn is a convolutional self-attention neural network, and MSA is a multi-head self-attention neural network.

[0071] Among them, the convolutional self-attention neural network ConvAttn can be expressed as:

[0072] ConvAttn(Di-1 )=σ(Conv2d k＝3,s＝1 (D i-1 ))ΘConv2d k＝3,s＝1 (D i-1 );

[0073] Among them, σ represents the sigmoid activation function, Θ represents the matrix element-level multiplication, Conv2d k＝3,s＝1 Represents a convolutional neural network with a kernel size of 3 and a stride of 1.

[0074] In the multi-head self-attention neural network MSA, three independent linear layers are first used to fusion the feature map D i-1 It is transformed into three multi-head tensors with the same size and number of heads, namely query (Q), keyword (K), and value (V). The output obtained by performing self-attention-based operations on each head of these three tensors is the final output of the multi-head self-attention neural network. The specific operation can be defined as:

[0075] MSA(Q, K, V) = [head1, head2,..., head h 】W 0 ;

[0076] head i =Attention(QW i Q , QW i K , QW i V );

[0077] Among them, h represents the total number of heads, all W(W 0 , W i Q , W i K , W i V ) represents the learnable parameters in the self-attention module. The definition of the self-attention mechanism can be the same as that of the attention mechanism.

[0078] In one embodiment, feature enhancement of the fused feature map output by the Nth cross-attention module may include the following steps: determining a learnable parameter according to the size of the fused feature map output by the Nth cross-attention module; and feature enhancement of the fused feature map output by the Nth cross-attention module according to the learnable parameter.

[0079] Taking N as 4 as an example, assuming that the size of the image to be enhanced is (256, 256, 3), the output sizes of the 1st to 4th encoding structures are: (128, 128, 64), (64, 64, 128), (32, 32, 320), (16, 16, 512); the output sizes of the 1st to 4th feature sequence modules are: (128, 128, 12), (64, 64, 48), (32, 32, 192), (16, 16, 768). According to the size of the fused feature map output by the 4th cross-attention module (16, 16, 512), it can be determined that the size of the learnable parameters that need to be incorporated into the underlying encoding structure is (1, 1, 512).

[0080] Specifically, enhancing the features of the fused feature map output by the Nth cross-attention module according to the learnable parameters may include: enhancing the features after adding the pixels of each spatial position point of the fused feature map output by the Nth cross-attention module to the learnable parameters.

[0081] In one embodiment, the fused feature map output by the fourth cross-attention module and the learnable parameters may be incorporated as follows: each spatial position point (i, j, 512) 0≦i, j≦15 on the fused feature map of size (16, 16, 512) is added to the learnable parameters to obtain a new feature map.

[0082] At present, the optimization direction of image enhancement models is usually to optimize the model for one or two image enhancement tasks, which reduces the universality and generalizability of the model to a certain extent. This embodiment uses an encoder based on a multi-level jump forward connection method to capture more image details. At the same time, a self-attention module is added between the encoder and the decoder, which can enhance the image features according to the image enhancement subtask category, and can also use learnable parameters to adaptively weight the underlying feature map, which is suitable for a variety of image enhancement tasks. Experiments have shown that the image enhancement network of this embodiment can cope with images collected by the vehicle-mounted vision system that are affected by various weather or environmental factors, effectively improving the safety and stability of the vehicle-mounted intelligent driving system.

[0083] Figure 5 FIG. 4 shows a flow chart of an image enhancement method in an embodiment. Figure 5 As shown, an image enhancement method is provided, which may include the following steps:

[0084] Step 501, obtaining an image to be enhanced;

[0085] Step 502: input the image to be enhanced into the above-mentioned image enhancement network to obtain an enhanced image output by the image enhancement network.

[0086] It should be understood that in some harsh weather conditions, the images captured by the on-board visual sensors of the intelligent driving system may have low image quality in part or all of the field of view, such as when the images are blocked by rain and snow, blocked by raindrops or dirt, or blurred by excessive rain or fog, etc. The above-mentioned image enhancement network can restore or enhance these low-quality images, thereby improving the stability and safety of the intelligent assisted driving system.

[0087] Specifically, the images collected by the vehicle-mounted sensors are transmitted to the image enhancement system in real time. The image enhancement system first determines whether the image is affected by the weather or environment. If it is determined that the image is not affected, it is directly transmitted to the corresponding vehicle-mounted intelligent driving application. If the image is affected by the weather or environment, such as showing some form of low-quality features, the image is determined as an image to be enhanced and transmitted to the image enhancement network for processing, and then the enhanced image is transmitted to the corresponding vehicle-mounted intelligent driving application.

[0088] The image enhancement network constructed by the multi-stage skip-forward coding structure in this embodiment can handle the visual enhancement problem in bad weather such as rain, snow, fog, muddy roads, etc., that is, it can complete image enhancement tasks such as image rain and snow removal, image raindrop or dirt removal, image deblurring, etc. for images captured by the visual sensor. Compared with the traditional autoencoder structure based on skip connection, the enhancement effect on various image enhancement tasks is improved.

[0089] The following describes an apparatus embodiment of the present application, which can be used to perform the image enhancement method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the image enhancement method of the present application.

[0090] See also Figure 6 , shows a block diagram of the image enhancement device in an embodiment of the present application.

[0091] like Figure 6 As shown, the image enhancement device of an embodiment of the present application includes: an image acquisition unit 601 and an image enhancement unit 602, wherein the image acquisition unit 601 is used to acquire the image to be enhanced; the image enhancement unit 602 is used to input the image to be enhanced into the above-mentioned image enhancement network to obtain the enhanced image output by the image enhancement network.

[0092] Based on the same inventive concept, the present application also provides an image enhancement device, referring to Figure 7, shows a schematic diagram of the structure of the image enhancement device in an embodiment of the present application, wherein the image enhancement device includes one or more memories 704, one or more processors 702, and at least one computer program (computer program instruction) stored in the memory 704 and executable on the processor 702, and the processor 702 implements the method described above when executing the computer program.

[0093] Among them, Figure 7 In the embodiment of the present invention, a bus architecture (represented by bus 700) is shown, which may include any number of interconnected buses and bridges, and bus 700 links various circuits including one or more processors represented by processor 702 and memory represented by memory 704. Bus 700 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 705 provides an interface between bus 700 and receiver 701 and transmitter 703. Receiver 701 and transmitter 703 may be the same element, namely a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 702 is responsible for managing bus 700 and general processing, while memory 704 may be used to store data used by processor 702 when performing operations.

[0094] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer program instruction is stored. The at least one computer program instruction is loaded and executed by a processor to implement the operations performed by the method described above.

[0095] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0097] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly or all or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store computer program instructions.

[0099] The above description is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for acquiring an image enhancement network, characterized in that: The image enhancement network includes: a self-attention module, N encoding modules, N feature sequence modules, N cross attention modules and N decoding modules, N is an integer greater than 1; each feature sequence module is connected in series, the feature sequence module corresponds to the cross attention module one by one, the output end of the feature sequence module is also connected to the input end of the cross attention module, the input data of the first feature sequence module is the image to be enhanced, the encoding module and the cross attention module are arranged at intervals and connected in series, the input data of the first encoding module is the image to be enhanced, the Nth cross attention module is connected to the first decoding module via the self-attention module, each decoding module is connected in series, the first N-1 encoding modules correspond to the last N-1 decoding modules one by one and are jump-connected; wherein, The feature sequence combination module is used to downsample the image to be enhanced or the feature map output by the previous feature sequence combination module to obtain a sequence feature map; The cross attention module is used to perform feature fusion on the combined sequence feature map and the feature map output by the previous encoding module based on the cross attention mechanism to obtain a fused feature map; The encoding module is used to extract features from the image to be enhanced or the fused feature map to obtain a third feature map; The self-attention module is used to enhance the features of the fused feature map output by the Nth cross-attention module to obtain an enhanced feature map; The decoding module is used to fuse the enhanced feature map or the third feature map with the image output by the previous decoding module and then decode it to obtain an enhanced image.

2. The method for acquiring an image enhancement network according to claim 1, characterized in that: The step of downsampling the image to be enhanced or the feature map output by the previous feature sequence combination module to obtain a sequenced feature map includes: Perform pixel traversal on the image to be enhanced or the feature map output by the previous feature sequence combination module; Arrange multiple pixels that are integer multiples of the pixel step length obtained by distance traversal to obtain a down-sampled image corresponding to the pixel; The downsampled images corresponding to all pixels in the feature map output by the previous feature sequence combination module or the image to be enhanced are arranged side by side to obtain the sequence combination feature map.

3. The method for acquiring an image enhancement network according to claim 2, characterized in that: The method of fusing the combined sequence feature map with the feature map output by the previous encoding module based on the cross attention mechanism to obtain a fused feature map includes: Performing a linear transformation on the combined sequence feature graph to obtain a first multi-head tensor and a second multi-head tensor; Performing a linear transformation on the feature map output by the previous encoding module to obtain a third multi-head tensor; Based on the cross attention mechanism, each head of the first multi-head tensor, the second multi-head tensor and the third multi-head tensor is operated to obtain the fused feature map.

4. The method for acquiring an image enhancement network according to claim 3, characterized in that: The attention mechanism is: Among them, Q is the first multi-head tensor, K is the second multi-head tensor, V is the third multi-head tensor, and d is the dimension of the feature in the multi-head tensor.

5. The method for acquiring an image enhancement network according to claim 1, characterized in that: The number N is 4, the first encoding module and the second encoding module both use convolutional self-attention neural networks, and the third encoding module and the fourth encoding module both use multi-head self-attention neural networks.

6. The method for acquiring an image enhancement network according to claim 5, characterized in that: The step of extracting features from the image to be enhanced or the fused feature map to obtain a third feature map includes: Down-sampling the image to be enhanced or the fused feature map using a convolutional neural network to obtain a down-fused feature map; The convolutional self-attention neural network or the multi-head self-attention neural network is used to perform feature extraction on the lower fusion feature map to obtain the third feature map.

7. The method for acquiring an image enhancement network according to any one of claims 1 to 6, characterized in that: The feature enhancement of the fused feature map output by the Nth cross attention module includes: Determining a learnable parameter according to the size of the fused feature map output by the Nth cross attention module; The fused feature map output by the Nth cross-attention module is feature enhanced according to the learnable parameters.

8. The method for acquiring an image enhancement network according to claim 7, characterized in that: The step of enhancing the features of the fused feature map output by the Nth cross attention module according to the learnable parameters includes: The pixel of each spatial position point of the fusion feature map output by the Nth cross-attention module is added to the learnable parameter and then the feature is enhanced.

9. An image enhancement method, characterized in that: include: Acquire an image to be enhanced; The image to be enhanced is input into the image enhancement network according to any one of claims 1 to 8 to obtain an enhanced image output by the image enhancement network.

10. An image enhancement device, comprising a processor and a memory, characterized in that: The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the instructions of the method according to claim 9 are implemented.

Images