Super-resolution method and device of image, storage medium and electronic device

Through the image super-resolution method, the residual group module is used to generate high-frequency feature maps, which solves the problem of poor image quality of the on-board camera in low light or long distance, improves image resolution and clarity, and enhances driving assistance functions.

CN118967441APending Publication Date: 2024-11-15CHONGQING CHANGAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410991391.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Car cameras have poor image quality and low resolution at long distances or low light conditions, making it difficult to provide high-quality driving assistance functions.

Method used

The image super-resolution method is adopted to obtain low-resolution images acquired by the camera, extract shallow features, and use the residual group module to generate high-frequency feature maps to reconstruct high-resolution images to improve image resolution.

Benefits of technology

It improves the clarity and resolution of the image, enhances the visualization ability of the vehicle during driving, assists in target recognition and detection, and improves driving judgment ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967441A_ABST
    Figure CN118967441A_ABST
Patent Text Reader

Abstract

The invention provides an image super-resolution method and device, a storage medium and an electronic device.The method comprises the steps that an original image collected by a camera is obtained, and the resolution of the original image is lower than a preset resolution threshold value; extracting shallow layer features of the original image; a residual group module is adopted to generate a high-frequency characteristic pattern of the shallow layer characteristics; and reconstructing and generating a high-resolution image of the original image by using the high-frequency feature map, the resolution of the high-resolution image being higher than a preset resolution threshold. According to the embodiment of the invention, clearer and more detailed image information is provided for a user, and the problems of poor quality and low resolution of an image acquired by a camera under an extreme condition in the prior art are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image super-resolution method and device, a storage medium, and an electronic device. Background Art

[0002] In the related art, vehicle-mounted cameras are usually used to provide driving assistance functions, such as driving records, blind spot detection, etc. In vehicle-mounted camera applications, image resolution is critical to accurately capture details and provide high-quality images. However, due to hardware and bandwidth limitations, vehicle-mounted cameras are usually limited by resolution, resulting in poor image quality captured by the camera at long distances or in low light conditions. Therefore, a method is needed to improve image resolution through software algorithm processing, thereby improving the visualization capability during vehicle driving.

[0003] In recent years, deep learning super-resolution technology has been widely used, but its application in vehicle-mounted camera image processing tasks is relatively rare. Unlike the above two solutions, the solution of the present invention performs super-resolution processing on the original image captured by the camera, which can not only improve the quality and resolution of the image, so that the vehicle can "see more clearly" during driving, but also assist downstream tasks, such as target recognition or detection, thereby improving the vehicle's "judgment" ability.

[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0005] The present invention provides an image super-resolution method and device, a storage medium, and an electronic device to solve the technical problems in related technologies.

[0006] According to one embodiment of the present invention, there is provided a method for super-resolution of an image, comprising: acquiring an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold; extracting shallow features of the original image; using a residual group module to generate a high-frequency feature map of the shallow features; and using the high-frequency feature map to reconstruct a high-resolution image of the original image, wherein the resolution of the high-resolution image is higher than the preset resolution threshold.

[0007] Optionally, extracting shallow features of the original image includes: determining the following dimensional information of the original image: width, height, and number of channels; inputting the dimensional information into a 3×3 two-dimensional convolutional network, and outputting the shallow features of the original image.

[0008] Optionally, using a residual group module to generate a high-frequency feature map of the shallow features includes: inputting the shallow features into multiple consecutive residual group modules and outputting residual features; inputting the residual features into a convolutional layer and a long skip connection and outputting a high-frequency feature map.

[0009] Optionally, the shallow features are input into multiple consecutive residual group modules, and the output residual features include: for each residual group module, the shallow features are input into the starting SCAM, and the starting data is output, wherein each residual group module includes: multiple spatial channel attention modules SCAM, 3×3 two-dimensional convolution, and a residual jump connection, and the multiple SCAMs include a starting SCAM, several intermediate SCAMs, and an ending SCAM arranged in sequence; the output data of the previous SCAM is used as the input data of the next SCAM, passing through the several intermediate SCAMs in sequence until the ending SCAM outputs the ending data; the 3×3 two-dimensional convolution is used to extract the high-frequency features of the ending data, and the residual features are output through the residual jump connection.

[0010] Optionally, the shallow feature is input into a starting SCAM, and outputting starting data includes: using a convolution layer and a ReLU activation layer to extract a first deep feature of the shallow feature, wherein each SCAM includes: a spatial attention module SAM, a channel attention module CAM, and an adaptive weighted fusion module AWF; inputting the first deep feature into the SAM, and extracting the spatial feature of the first deep feature, and inputting the first deep feature into the CAM, and extracting the hierarchical feature of the first deep feature; inputting the hierarchical feature and the spatial feature into the AWF, and outputting a fused feature; using a convolution layer to integrate the fused feature, and performing a jump connection between the integrated feature and the first deep feature to obtain starting data.

[0011] Optionally, inputting the first deep feature into the SAM, and extracting the spatial feature of the first deep feature includes: using a convolution layer and a ReLU activation layer to extract a second deep feature of the first deep feature; inputting the second deep feature into a maximum pooling layer and an average pooling layer respectively, using the maximum pooling layer and the average pooling layer to perform feature dimension reduction on the second deep feature according to the channel dimension, and splicing the two results of the dimension reduction according to the channel dimension to obtain a spliced ​​feature; using 1×1 convolution and Sigmad activation function in sequence to extract a spatial weight map of the spliced ​​feature, and weighting the spatial weight map with the first deep feature to obtain a spatial feature.

[0012] Optionally, the deep features are input into the CAM, and the hierarchical features of the deep features are extracted, including: using a convolutional layer and a ReLU activation layer to extract a third deep feature of the first deep feature; inputting the third deep feature into a maximum pooling layer and an average pooling layer respectively, using the maximum pooling layer and the average pooling layer to perform feature dimensionality reduction on the third deep feature according to the spatial dimension, and inputting the two results of the dimensionality reduction into the MLP fully connected layer for feature mapping, respectively, to obtain two one-dimensional feature maps; adding the two one-dimensional feature maps and inputting them into a Sigmoid activation function to obtain a channel weight map, and multiplying the channel weight map by the first deep feature according to the channel to obtain a hierarchical feature.

[0013] Optionally, the hierarchical features and the spatial features are input into the AWF, and the output fused features include: inputting the first deep features into the AWF to generate a first weight vector and a second weight vector, wherein the AWF includes an average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence; using the first weight vector to weight the hierarchical features to obtain a first weighted result, and using the second weight vector to weight the spatial features to obtain a second weighted result; adding the first weighted result and the second weighted result to obtain the fused feature.

[0014] According to another embodiment of the present invention, there is provided a super-resolution device for an image, comprising: an acquisition module, used to acquire an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold; an extraction module, used to extract shallow features of the original image; a first generation module, used to generate a high-frequency feature map of the shallow features by using a residual group module; and a second generation module, used to reconstruct a high-resolution image of the original image by using the high-frequency feature map, wherein the resolution of the high-resolution image is higher than the preset resolution threshold.

[0015] Optionally, the acquisition module includes: a determination unit, used to determine the following dimensional information of the original image: width, height, and number of channels; and a processing unit, used to input the dimensional information into a 3×3 two-dimensional convolutional network and output shallow features of the original image.

[0016] Optionally, the first generation module includes: a first processing unit, used to input the shallow features into multiple consecutive residual group modules and output residual features; a second processing unit, used to input the residual features into a convolutional layer and a long skip connection, and output a high-frequency feature map.

[0017] Optionally, the first processing unit includes: a first processing sub-unit, used to input the shallow features into the starting SCAM and output the starting data for each residual group module, wherein each residual group module includes: multiple spatial channel attention modules SCAM, 3×3 two-dimensional convolution, and a residual jump connection, and the multiple SCAMs include a starting SCAM, several intermediate SCAMs, and an ending SCAM arranged in sequence; a second processing sub-unit, used to use the output data of the previous SCAM as the input data of the next SCAM, passing through the several intermediate SCAMs in sequence until the ending SCAM outputs the ending data; an output unit, used to use the 3×3 two-dimensional convolution to extract the high-frequency features of the ending data, and output the residual features through the residual jump connection.

[0018] Optionally, the first processing subunit is also used to: use a convolution layer and a ReLU activation layer to extract the first deep feature of the shallow feature, wherein each SCAM includes: a spatial attention module SAM, a channel attention module CAM, and an adaptive weighted fusion module AWF; input the first deep feature into the SAM, and extract the spatial feature of the first deep feature, and input the first deep feature into the CAM, and extract the hierarchical feature of the first deep feature; input the hierarchical feature and the spatial feature into the AWF, and output the fused feature; use a convolution layer to integrate the fused feature, and jump-connect the integrated feature with the first deep feature to obtain the starting data.

[0019] Optionally, the first processing subunit is also used to: use a convolution layer and a ReLU activation layer to extract a second deep feature of the first deep feature; input the second deep feature into a maximum pooling layer and an average pooling layer respectively, use the maximum pooling layer and the average pooling layer to perform feature dimensionality reduction on the second deep feature according to the channel dimension, and splice the two results of the dimensionality reduction according to the channel dimension to obtain a spliced ​​feature; use 1×1 convolution and Sigmoid activation function in sequence to extract a spatial weight map of the spliced ​​feature, and weight the spatial weight map with the first deep feature to obtain a spatial feature.

[0020] Optionally, the first processing subunit is also used to: use a convolution layer and a ReLU activation layer to extract a third deep feature of the first deep feature; input the third deep feature into a maximum pooling layer and an average pooling layer respectively, use the maximum pooling layer and the average pooling layer to perform feature dimensionality reduction on the third deep feature according to the spatial dimension, and input the two results of the dimensionality reduction into the MLP fully connected layer for feature mapping to obtain two one-dimensional feature maps; add the two one-dimensional feature maps and input them into a Sigmoid activation function to obtain a channel weight map, multiply the channel weight map by the first deep feature according to the channel to obtain a hierarchical feature.

[0021] Optionally, the first processing subunit is also used to: input the first deep feature into AWF to generate a first weight vector and a second weight vector, wherein the AWF includes an average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence; use the first weight vector to weight the hierarchical features to obtain a first weighted result, and use the second weight vector to weight the spatial features to obtain a second weighted result; add the first weighted result and the second weighted result to obtain a fused feature.

[0022] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0023] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0024] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0025] Beneficial effects of the present invention:

[0026] 1. Process the low-resolution images captured by the camera through software algorithms to improve image quality and resolution, thereby providing users with clearer and more detailed image information;

[0027] 2. The SCAM module with a specific structure is used to weigh the importance of the two branch outputs, space and channel, and can output more accurate high-frequency feature maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 is a hardware structure block diagram of a car according to an embodiment of the present invention;

[0030] Figure 2 is a flow chart of an image super-resolution method according to an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of a spatial channel attention module in an embodiment of the present invention;

[0032] Figure 4 is a diagram of the spatial channel attention network architecture in an embodiment of the present invention;

[0033] Figure 5 is a flowchart of a solution in an embodiment of the present invention;

[0034] Figure 6 The present invention is a structural block diagram of an image super-resolution device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Example 1

[0038] The method embodiment provided in the first embodiment of the present application can be executed in an automobile, an automobile controller, a gear controller, an automobile processor or a similar processing device. Taking running on an automobile as an example, Figure 1 1 is a hardware structure diagram of a car according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0039] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a remote control method for a car in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0041] In this embodiment, a super-resolution method for an image is provided. Figure 2 is a flow chart of a super-resolution method for an image according to an embodiment of the present invention. Figure 2As shown, the process includes the following steps:

[0042] Step S202, obtaining an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold;

[0043] The camera of this embodiment may be a vehicle-mounted camera, or a camera integrated on other devices, or a separate camera device.

[0044] Step S204, extracting shallow features of the original image;

[0045] Optionally, the shallow features include pixel information and fine-grained details, which may be, but are not limited to, feature information such as color, texture, edge, and corner of the original image. The shallow features are used to characterize the detail information of the original image.

[0046] Step S206, using a residual group module to generate a high-frequency feature map of the shallow features;

[0047] Optionally, the high-frequency feature map is an image whose grayscale variation is greater than a preset threshold.

[0048] Step S208: Reconstructing and generating a high-resolution image of the original image using the high-frequency feature map, wherein the resolution of the high-resolution image is higher than a preset resolution threshold.

[0049] During the reconstruction process, the high-frequency feature maps extracted by the network will be super-resolved. This process is completed by an upsampling module and a convolutional layer. The upsampling module is used to improve the resolution of the image, while the convolutional layer fuses the information of the upsampling results, and finally a high-resolution image can be obtained.

[0050] The extracted high-frequency features f HR Input into the super-resolution reconstruction module to obtain the final high-resolution image HR = Conv 3×3 (UpSampling(f HR )), where UpSampling(·) represents an upsampling operation.

[0051] Through the above steps, an original image captured by a camera is obtained, wherein the resolution of the original image is lower than a preset resolution threshold; shallow features of the original image are extracted; a high-frequency feature map of the shallow features is generated by a residual group module; a high-resolution image of the original image is reconstructed by using the high-frequency feature map, wherein the resolution of the high-resolution image is higher than the preset resolution threshold, a high-frequency feature map is generated by using shallow features of the low-resolution original image, and a high-resolution image is reconstructed, and the low-resolution image captured by the camera is processed by a software algorithm to improve the image quality and resolution, thereby providing users with clearer and more detailed image information, and solving the problem of poor image quality and low resolution of images captured by the camera under extreme conditions in the related art.

[0052] In one implementation of this embodiment, extracting the shallow features of the original image includes: determining the following dimensional information of the original image: width, height, and number of channels; inputting the dimensional information into a 3×3 two-dimensional convolutional network, and outputting the shallow features of the original image.

[0053] The original image captured by the camera is subjected to a 3×3 two-dimensional convolution to extract shallow features in the image. This operation can be expressed as f LR =Conv 3×3 (LR), where f LR Represents the shallow feature map extracted from the input low-resolution image (image representing shallow features), Conv 3×3 (·) represents a 3×3 two-dimensional convolution, and LR represents the input low-resolution original image.

[0054] The input low-resolution original image (LR) dimension is set to W×H×C, where W represents the width of the image, H represents the height of the image, and C represents the number of channels; first, the shallow feature f is extracted through the convolution layer LR =Conv 3×3 (LR).

[0055] In this embodiment, using the residual group module to generate the high-frequency feature map of the shallow feature includes: inputting the shallow feature into multiple consecutive residual group modules and outputting residual features; inputting the residual features into the convolution layer and the long skip connection and outputting the high-frequency feature map.

[0056] In one example, the number of residual group modules is four. After the first layer of convolution extracts shallow features, it will be input into the subsequent four residual group modules (RG). After four consecutive RG modules, the output result will be extracted through a 3×3 two-dimensional convolution and a long-distance jump connection to extract the high-frequency feature map in the input image and input it into the subsequent reconstruction task.

[0057] In one example, the shallow features are input into multiple consecutive residual group modules, and the output residual features include: for each residual group module, the shallow features are input into the starting SCAM, and the starting data is output, wherein each residual group module includes: multiple spatial channel attention modules SCAM, 3×3 two-dimensional convolution, and a residual jump connection, and the multiple SCAMs include a starting SCAM, several intermediate SCAMs, and an ending SCAM arranged in sequence; the output data of the previous SCAM is used as the input data of the next SCAM, passing through the several intermediate SCAMs in sequence until the ending SCAM outputs the ending data; the 3×3 two-dimensional convolution is used to extract the high-frequency features of the ending data, and the residual features are output through the residual jump connection.

[0058] Taking the i-th RG module as an example, the output of the i-th RG module is RG(f i ) = f i +Conv 3×3 (SCAM4(…SCAM1(f i ))), where f i represents the input of the i-th RG module, SCAM i (·) represents the output of the i-th SCAM module in the RG module; the input feature map of the SCAM module will be extracted through two convolutional layers and a ReLU activation function. i =Conv 3×3 (RelU(Conv 3×3 (f i ))). Taking the number of SCAMs as four as an example, each residual group module is composed of four spatial channel attention modules (SCAMs), a 3×3 two-dimensional convolution, and a residual skip connection. The shallow features are passed through four consecutive RG modules, a convolution layer, and a long skip connection to focus on learning the high-frequency features in the image. HR =f LR +Conv 3×3 (RG4(…RG1(f LR ))), where f HR Represents the extracted high-frequency feature map, RG i (·) represents the output result of the i-th RG module.

[0059] In RG, the output of each SCAM module will be used as the input of the next SCAM, and the output of the last SCAM module will be used to extract medium and high frequency features through 3×3 two-dimensional convolution. Finally, a residual jump connection is used to prevent the gradient from disappearing due to excessive network depth, and to further compensate for the information loss caused by the SCAM module.

[0060] In the above example, the shallow feature is input into the starting SCAM, and the output starting data includes: using a convolution layer and a ReLU activation layer to extract the first deep feature of the shallow feature, wherein each SCAM includes: a spatial attention module SAM, a channel attention module CAM, and an adaptive weighted fusion module AWF; inputting the first deep feature into the SAM, and extracting the spatial feature of the first deep feature, and inputting the first deep feature into the CAM, and extracting the hierarchical feature of the first deep feature; inputting the hierarchical feature and the spatial feature into the AWF, and outputting the fused feature; using a convolution layer to integrate the fused feature, and performing a jump connection between the integrated feature and the first deep feature to obtain the starting data.

[0061] For each SCAM module, it contains a spatial attention module (SAM), a channel attention module (CAM), and an adaptive weighted fusion module (AWF). First, the input of SCAM will pass through two 3×3 convolutional layers and a ReLU activation layer to extract deep features. Then the SAM and CAM modules will filter and process the spatial features and hierarchical features respectively. Finally, the AWF module will fuse the output results of SAM and CAM to further weigh the importance of the outputs of the two branches, and then a convolutional layer will be used to integrate the features to obtain the output data (feature map) of the entire SCAM module. Figure 3 : is a schematic diagram of the spatial channel attention module in an embodiment of the present invention, the input feature map corresponds to the shallow feature, the output feature map corresponds to the residual feature, the spatial attention branch corresponds to SAM, and the channel attention branch corresponds to CAM. represents weighted multiplication, and ⊕ represents addition.

[0062] The extracted features x i (Shallow features) will be processed by the spatial attention module SAM, the channel attention module CAM and the adaptive weighted fusion module AWF, so the output of the i-th SCAM is Where SAM(x i ) and CAM(x i ) represent the output results (spatial features and hierarchical features) of the SAM and CAM modules in the i-th SCAM module, respectively, and and Represents the two weight vectors output by the AWF module.

[0063] In one embodiment, the first deep feature is input into the SAM, and the spatial feature of the first deep feature is extracted, including: using a convolution layer and a ReLU activation layer to extract a second deep feature of the first deep feature; inputting the second deep feature into a maximum pooling layer and an average pooling layer respectively, using the maximum pooling layer and the average pooling layer to perform feature dimension reduction on the second deep feature according to the channel dimension, and splicing the two results of the dimension reduction according to the channel dimension to obtain a spliced ​​feature; using 1×1 convolution and Sigmoid activation function in sequence to extract a spatial weight map of the spliced ​​feature, and weighting the spatial weight map with the first deep feature to obtain a spatial feature.

[0064] The SAM module is used to focus on the effective information in each feature map in the spatial dimension, while giving less weight to unimportant spatial information. The input of SAM is first subjected to deep feature extraction by the convolution layer and the ReLU activation layer. Then, the input is subjected to feature dimensionality reduction by the channel dimension using the maximum pooling and average pooling respectively, and the two results of dimensionality reduction are concatenated by the channel dimension. Finally, the spatial weight map is obtained through 1×1 convolution and Sigmoid activation function, and is weighted with the input. Through this process, important information in the input feature space can be enhanced and unimportant information can be weakened.

[0065] For the SAM module, its input W×H×64 is first processed by the convolutional layer and the ReLU activation function to further extract the feature s i =Conv 3×3 (RelU(x i )), and then reduce the dimension by channel dimension through maximum pooling and average pooling, and the corresponding dimensions are W×H×1. Finally, concatenate the channel dimension to W×H×2, and use the spatial weight SAM(x i )=Sigmoid(Conv 3×3 (Concat(MaxPooling(s i ),AveragePooling(s i )))).

[0066] In one embodiment, the deep features are input into the CAM, and the hierarchical features of the deep features are extracted, including: using a convolutional layer and a ReLU activation layer to extract a third deep feature of the first deep feature; inputting the third deep feature into a maximum pooling layer and an average pooling layer respectively, using the maximum pooling layer and the average pooling layer to perform feature dimensionality reduction on the third deep feature according to the spatial dimension, and inputting the two results of the dimensionality reduction into the MLP fully connected layer for feature mapping, respectively, to obtain two one-dimensional feature maps; adding the two one-dimensional feature maps and inputting them into a Sigmoid activation function to obtain a channel weight map, and multiplying the channel weight map with the first deep feature according to the channel to obtain a hierarchical feature.

[0067] The CAM module is used to focus on the importance of feature maps in the channel dimension, giving larger weights to important features and smaller weights to unimportant feature maps, so as to screen out more useful features for reconstructing super-resolution images. Specifically, the input of CAM will pass through the convolution layer and ReLU activation layer to further extract features, and then the maximum pooling and average pooling will be used to reduce the dimension according to the spatial dimension, and the results of the dimension reduction will be input into the MLP fully connected layer for feature mapping. Finally, the two one-dimensional feature maps are added and input into the Sigmoid activation function to obtain the weight in the channel dimension, and then the weight is multiplied by the input feature according to the channel to complete the feature map screening task.

[0068] The input of CAM will also be extracted by convolution layer and ReLU activation function. i =Conv 3×3 (RelU(x i )), and then the spatial dimension is reduced by maximum pooling and average pooling, and the feature mapping is performed using the MLP fully connected layer, and the corresponding dimensions are 1×1×64. Finally, the two outputs are added and then the channel weight CAM(x i )=Sigmoid(MLP(MaxPooling(c i ))+MLP(AveragePooling(c i ))).

[0069] In one embodiment, the hierarchical features and the spatial features are input into the AWF, and the output fused features include: inputting the first deep features into the AWF to generate a first weight vector and a second weight vector, wherein the AWF includes an average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence; using the first weight vector to weight the hierarchical features to obtain a first weighted result, and using the second weight vector to weight the spatial features to obtain a second weighted result; adding the first weighted result and the second weighted result to obtain the fused feature.

[0070] In order to better integrate the output results of the SAM and CAM modules, AWF generates weight vectors α and β (corresponding to the first weight vector and the second weight vector) through average pooling, two fully connected layers, ReLU activation function and Sigmoid activation function for the input feature map, where the α vector is responsible for weighting the output results of the CAM module, and the β vector is responsible for weighting the output results of the SAM module, and further add the weighted results of the two as the output of the AWF module. Finally, the output of AWF is feature integrated through the convolution layer and jump-connected with the input feature map, so as to obtain the output result of the SCAM module.

[0071] For the AWF module, it will enter x i Through a series of operations, two dynamic weights are obtained

[0072] The fully connected layer is implemented using 1×1 convolution, which can effectively reduce the amount of network parameters.

[0073] Figure 4 is a diagram of the spatial channel attention network architecture in an embodiment of the present invention. Based on this network architecture, Figure 5It is a flowchart of the scheme in the embodiment of the present invention, and the process includes two stages: one is to collect a large amount of vehicle-mounted camera image data for training the spatial channel attention network model; the other is to apply the trained network model to the vehicle-mounted camera for image processing. For stage two, see the description of the above embodiment. For stage one, it is first necessary to collect a large number of high- and low-resolution image pairs with different scenes, and then enhance or pre-process these images and divide them into training images, verification images and test images; during the training process, the training images are used to train the constructed spatial channel attention network, and the verification images are used to verify whether the network model has converged during the training process. If it has converged, then the training is terminated and the model is saved; the trained network model is used for the test data set to test the generalization ability of the network. If the expected effect is achieved, then the network model can be deployed in the vehicle-mounted camera to solve the problem of low resolution of the collected image in extreme cases. Otherwise, it is necessary to adjust the network parameters, such as the network learning rate, the number of iterations, etc., and scramble the data and retrain.

[0074] By adopting the solution of this embodiment, the low-resolution images captured by the camera are processed through a software algorithm to improve the image quality and resolution, thereby providing users with clearer and more detailed image information, and can solve the problem of poor image quality and low resolution captured by the vehicle-mounted camera under extreme conditions.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0076] Example 2

[0077] In this embodiment, a super-resolution device for an image is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0078] Figure 6 is a structural block diagram of an image super-resolution device according to an embodiment of the present invention, such as Figure 6As shown, the device comprises:

[0079] An acquisition module 60 is used to acquire an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold;

[0080] An extraction module 62, used to extract shallow features of the original image;

[0081] A first generating module 64, configured to generate a high-frequency feature map of the shallow features by using a residual group module;

[0082] The second generating module 66 is used to reconstruct a high-resolution image of the original image using the high-frequency feature map, wherein the resolution of the high-resolution image is higher than a preset resolution threshold.

[0083] Optionally, the acquisition module includes: a determination unit, used to determine the following dimensional information of the original image: width, height, and number of channels; and a processing unit, used to input the dimensional information into a 3×3 two-dimensional convolutional network and output shallow features of the original image.

[0084] Optionally, the first generation module includes: a first processing unit, used to input the shallow features into multiple consecutive residual group modules and output residual features; a second processing unit, used to input the residual features into a convolutional layer and a long skip connection, and output a high-frequency feature map.

[0085] Optionally, the first processing unit includes: a first processing sub-unit, used to input the shallow features into the starting SCAM and output the starting data for each residual group module, wherein each residual group module includes: multiple spatial channel attention modules SCAM, 3×3 two-dimensional convolution, and a residual jump connection, and the multiple SCAMs include a starting SCAM, several intermediate SCAMs, and an ending SCAM arranged in sequence; a second processing sub-unit, used to use the output data of the previous SCAM as the input data of the next SCAM, passing through the several intermediate SCAMs in sequence until the ending SCAM outputs the ending data; an output unit, used to use the 3×3 two-dimensional convolution to extract the high-frequency features of the ending data, and output the residual features through the residual jump connection.

[0086] Optionally, the first processing subunit is also used to: use a convolution layer and a ReLU activation layer to extract the first deep feature of the shallow feature, wherein each SCAM includes: a spatial attention module SAM, a channel attention module CAM, and an adaptive weighted fusion module AWF; input the first deep feature into the SAM, and extract the spatial feature of the first deep feature, and input the first deep feature into the CAM, and extract the hierarchical feature of the first deep feature; input the hierarchical feature and the spatial feature into the AWF, and output the fused feature; use a convolution layer to integrate the fused feature, and jump-connect the integrated feature with the first deep feature to obtain the starting data.

[0087] Optionally, the first processing subunit is also used to: use a convolution layer and a ReLU activation layer to extract a second deep feature of the first deep feature; input the second deep feature into a maximum pooling layer and an average pooling layer respectively, use the maximum pooling layer and the average pooling layer to perform feature dimensionality reduction on the second deep feature according to the channel dimension, and splice the two results of the dimensionality reduction according to the channel dimension to obtain a spliced ​​feature; use 1×1 convolution and Sigmoid activation function in sequence to extract a spatial weight map of the spliced ​​feature, and weight the spatial weight map with the first deep feature to obtain a spatial feature.

[0088] Optionally, the first processing subunit is also used to: use a convolution layer and a ReLU activation layer to extract a third deep feature of the first deep feature; input the third deep feature into a maximum pooling layer and an average pooling layer respectively, use the maximum pooling layer and the average pooling layer to perform feature dimensionality reduction on the third deep feature according to the spatial dimension, and input the two results of the dimensionality reduction into the MLP fully connected layer for feature mapping to obtain two one-dimensional feature maps; add the two one-dimensional feature maps and input them into a Sigmoid activation function to obtain a channel weight map, multiply the channel weight map by the first deep feature according to the channel to obtain a hierarchical feature.

[0089] Optionally, the first processing subunit is also used to: input the first deep feature into AWF to generate a first weight vector and a second weight vector, wherein the AWF includes an average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence; use the first weight vector to weight the hierarchical features to obtain a first weighted result, and use the second weight vector to weight the spatial features to obtain a second weighted result; add the first weighted result and the second weighted result to obtain a fused feature.

[0090] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0091] Example 3

[0092] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0093] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0094] S1, obtaining an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold;

[0095] S2, extracting shallow features of the original image;

[0096] S3, using a residual group module to generate a high-frequency feature map of the shallow features;

[0097] S4, reconstructing and generating a high-resolution image of the original image using the high-frequency feature map, wherein a resolution of the high-resolution image is higher than a preset resolution threshold.

[0098] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0099] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0100] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0101] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0102] S1, obtaining an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold;

[0103] S2, extracting shallow features of the original image;

[0104] S3, using a residual group module to generate a high-frequency feature map of the shallow features;

[0105] S4, reconstructing and generating a high-resolution image of the original image using the high-frequency feature map, wherein a resolution of the high-resolution image is higher than a preset resolution threshold.

[0106] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0110] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A super-resolution method for an image, characterized in that: include: Acquire an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold; Extracting shallow features of the original image; A residual group module is used to generate a high-frequency feature map of the shallow features; The high-frequency feature map is used to reconstruct and generate a high-resolution image of the original image, wherein the resolution of the high-resolution image is higher than the preset resolution threshold.

2. The method according to claim 1, characterized in that Extracting shallow features of the original image includes: Determine the following dimensional information of the original image: width, height, and number of channels; The dimensional information is input into a 3×3 two-dimensional convolutional network, and the shallow features of the original image are output.

3. The method according to claim 1, characterized in that The high-frequency feature map of the shallow features generated by the residual group module includes: Inputting the shallow features into a plurality of consecutive residual group modules, and outputting residual features; The residual features are input into the convolutional layer and the long skip connection, and a high-frequency feature map is output.

4. The method according to claim 3, characterized in that The shallow features are input into a plurality of consecutive residual group modules, and the output residual features include: For each residual group module, the shallow feature is input into the starting SCAM, and the starting data is output, wherein each residual group module includes: a plurality of spatial channel attention modules SCAM, a 3×3 two-dimensional convolution, and a residual jump connection, and the plurality of SCAMs include a starting SCAM, a plurality of intermediate SCAMs, and an ending SCAM arranged in sequence; Using the output data of the previous SCAM as the input data of the next SCAM, passing through the plurality of intermediate SCAMs in sequence, until the end SCAM outputs the end data; The 3×3 two-dimensional convolution is used to extract the high-frequency features of the end data, and the residual features are output through the residual jump connection.

5. The method according to claim 4, characterized in that The shallow features are input into the starting SCAM, and the output starting data includes: A convolutional layer and a ReLU activation layer are used to extract the first deep features of the shallow features, wherein each SCAM includes: a spatial attention module SAM, a channel attention module CAM, and an adaptive weighted fusion module AWF; Inputting the first deep feature into the SAM and extracting the spatial feature of the first deep feature, and inputting the first deep feature into the CAM and extracting the hierarchical feature of the first deep feature; Input the hierarchical features and the spatial features into the AWF, and output fusion features; The fused features are integrated using a convolutional layer, and the integrated features are jump-connected with the first deep features to obtain starting data.

6. The method according to claim 5, characterized in that Inputting the first deep feature into the SAM and extracting the spatial feature of the first deep feature comprises: Extracting the second deep features of the first deep features using a convolutional layer and a ReLU activation layer; Inputting the second deep features into the maximum pooling layer and the average pooling layer respectively, performing feature dimensionality reduction on the second deep features according to the channel dimension by using the maximum pooling layer and the average pooling layer, and splicing the two dimensionality reduction results according to the channel dimension to obtain a spliced ​​feature; A 1×1 convolution and a Sigmoid activation function are sequentially used to extract a spatial weight map of the concatenated feature, and the spatial weight map is weighted with the first deep feature to obtain a spatial feature.

7. The method according to claim 5, characterized in that Inputting the deep features into the CAM and extracting the hierarchical features of the deep features includes: A convolutional layer and a ReLU activation layer are used to extract a third deep feature of the first deep feature; Input the third deep features into the maximum pooling layer and the average pooling layer respectively, use the maximum pooling layer and the average pooling layer to perform feature dimension reduction on the third deep features according to the spatial dimension, and input the two results of dimension reduction into the MLP fully connected layer for feature mapping, so as to obtain two one-dimensional feature maps; The two one-dimensional feature maps are added and input into a Sigmoid activation function to obtain a channel weight map, and the channel weight map is multiplied by the first deep feature according to the channel to obtain a hierarchical feature.

8. The method according to claim 5, characterized in that The hierarchical features and the spatial features are input into the AWF, and the output fusion features include: Inputting the first deep feature into an AWF to generate a first weight vector and a second weight vector, wherein the AWF includes an average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence; The hierarchical features are weighted using the first weight vector to obtain a first weighted result, and the spatial features are weighted using the second weight vector to obtain a second weighted result; The first weighted result and the second weighted result are added together to obtain a fusion feature.

9. An image super-resolution device, characterized in that: include: An acquisition module, used to acquire an original image captured by a camera, wherein the resolution of the original image is lower than a preset resolution threshold; An extraction module, used for extracting shallow features of the original image; A first generating module, used for generating a high-frequency feature map of the shallow features by using a residual group module; The second generating module is used to reconstruct a high-resolution image of the original image by using the high-frequency feature map, wherein the resolution of the high-resolution image is higher than the preset resolution threshold.

10. The device according to claim 9, characterized in that The first generation module comprises: A first processing unit, configured to input the shallow features into a plurality of consecutive residual group modules and output residual features; The second processing unit is used to input the residual feature into the convolution layer and the long skip connection, and output a high-frequency feature map.

11. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.