Image processing method, electronic device and storage medium

Abstract

The disclosed embodiment discloses an image processing method, an electronic device and a storage medium, wherein the method comprises: inputting an image to be identified into a first convolutional neural network to obtain a quality reference value of the image to be identified; comparing the quality reference value with a first set threshold value and a second set threshold value; when the quality reference value is not less than the second set threshold value, determining that the image to be identified is a valid image; when the quality reference value is greater than the first set threshold value and less than the second set threshold value, determining that the image to be identified is a reconstructible image; and performing super-resolution reconstruction on the reconstructible image, wherein the number of super-resolution reconstructions performed on the reconstructible image does not exceed a set number. The disclosed embodiment can achieve channel weighting and pixel weighting of features before convolution of the convolution layer, which helps the convolution layer to perform better convolution processing on the features and improve the recognition accuracy, and can perform super-resolution reconstruction on the image, thereby improving the success rate of image recognition to a certain extent.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an electronic device, and a storage medium. Background Art

[0002] As online transactions (such as real estate online transactions) and online office services gradually enter a mature stage, the online recognition and review of various certification documents such as identity cards and real estate certificates are also showing a trend of continuous development. In the existing online recognition and review process of certification documents, users need to capture and upload images of certification documents through a camera device, and then upload the images to an online platform, which collects information about the certification documents through image recognition.

[0003] The existing online identification and review process for certification documents, due to the differences in camera devices and image distortion during the upload process, will lead to deviations in the collection of information when the online platform collects the certification document information through images. The certification document information still needs to be filled in through manual entry, which increases labor and time costs. Summary of the invention

[0004] A technical problem to be solved by the embodiments of the present disclosure is to provide an image processing method, an electronic device and a storage medium.

[0005] According to one aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0006] Inputting the image to be recognized into a first convolutional neural network to obtain a quality reference value of the image to be recognized, wherein the first convolutional neural network is provided with a dual attention module after at least one pooling layer and before at least one group of convolutional layers, and the dual attention module is used to perform channel weighting and pixel weighting on the feature map to be convolved by the at least one group of convolutional layers;

[0007] comparing the quality reference value with a first set threshold and a second set threshold;

[0008] When the quality reference value is not less than a second set threshold, determining that the image to be identified is a valid image;

[0009] When the quality reference value is greater than a first set threshold and less than the second set threshold, determining that the image to be identified is a reconstructible image;

[0010] The reconstructable image is super-resolution reconstructed, and the super-resolution reconstructed image is input into the first convolutional neural network, wherein the number of times the super-resolution reconstruction is performed on the image to be identified does not exceed a set number.

[0011] In one embodiment of the present disclosure, the process of the dual attention module processing the feature map that needs to be convolved by the at least one group of convolutional layers includes:

[0012] Performing global pooling on the feature maps that need to be convolved in the at least one group of convolutional layers to obtain a first weight set of channel dimensions;

[0013] Convolving the first weight set with a first set number of first convolution kernels to obtain a second weight set of channel dimension;

[0014] For the feature graphs that need to be convolved by the at least one group of convolutional layers, a weight of at least one pixel is calculated through an average pooling layer to obtain a third weight set of pixel dimensions;

[0015] Convolving the third weight set with a second set number of second convolution kernels to obtain a fourth weight set in pixel dimension;

[0016] The fourth weight set, the second weight set, and the feature map that needs to be convolved by the at least one group of convolutional layers are weighted to obtain a double-weighted feature map.

[0017] In yet another embodiment of the present disclosure, the first convolutional neural network is a pre-trained VGG-16 network;

[0018] And / or, the first convolution kernel is a 1*1 convolution kernel, and the second convolution kernel is a 3*3 convolution kernel.

[0019] In yet another embodiment of the present disclosure, the convolving the first weight set with a first set number of first convolution kernels to obtain a second weight set of channel dimension includes:

[0020] Convolving the first weight set with a first set number of first convolution kernels to obtain a weight value of at least one channel;

[0021] The weight value of the at least one channel that is less than the weight threshold is set to zero, and the weight value of the at least one channel that is not less than the weight threshold is retained to obtain a second weight set of the channel dimension.

[0022] In yet another embodiment of the present disclosure, the method further includes:

[0023] In response to the quality reference value being not greater than a second set threshold, determining that the image to be identified is an invalid image;

[0024] Outputting an invalid image prompt message, wherein the invalid image prompt message is used to prompt the user to upload a new image.

[0025] In yet another embodiment of the present disclosure, the super-resolution reconstruction of the reconstructible image includes:

[0026] Performing image super-resolution reconstruction on the reconstructable image using a second convolutional neural network, wherein the second convolutional neural network is a pre-trained fast super-high resolution convolutional neural network;

[0027] The feature extraction process when the second convolutional neural network performs image super-resolution reconstruction on the reconstructable image includes:

[0028] Using a third convolution kernel to perform channel grouping on the image to be identified;

[0029] The same convolution kernel is used to extract features from the groups obtained by channel grouping to obtain group features;

[0030] Performing mixed calculation on the grouping features to obtain mixed features;

[0031] Feature extraction is performed on the mixed features to obtain an image feature map.

[0032] In yet another embodiment of the present disclosure, the quality reference value of the image to be recognized is determined based on the number of output neurons of the normalization layer of the first convolutional neural network.

[0033] According to another aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0034] A quality determination module, used for inputting the image to be identified into a first convolutional neural network to obtain a quality reference value of the image to be identified, wherein the first convolutional neural network is provided with a dual attention module after at least one pooling layer and before at least one group of convolutional layers, and the dual attention module is used for performing channel weighting and pixel weighting on a feature map to be convolved by the at least one group of convolutional layers;

[0035] A comparison module, used for comparing the quality reference value with a first set threshold and a second set threshold;

[0036] A first determination module, configured to determine that the image to be identified is a valid image when the quality reference value is not less than a second set threshold;

[0037] A second determination module, configured to determine that the image to be identified is a reconstructible image when the quality reference value is greater than a first set threshold and less than a second set threshold;

[0038] A reconstruction module is used to perform super-resolution reconstruction on the reconstructible image and input the super-resolution reconstructed image into the first convolutional neural network, wherein the number of times the super-resolution reconstruction is performed on the image to be identified does not exceed a set number.

[0039] In one embodiment of the present disclosure, the quality determination module includes:

[0040] A global pooling submodule, used for performing global pooling on the feature maps that need to be convolved in the at least one group of convolutional layers to obtain a first weight set of channel dimensions;

[0041] A first convolution submodule, configured to convolve the first weight set through a first set number of first convolution kernels to obtain a second weight set of channel dimension;

[0042] An average pooling submodule, used for calculating the weight of at least one pixel of the feature map that needs to be convolved by the at least one group of convolutional layers through the average pooling layer, to obtain a third weight set of pixel dimensions, to obtain a third weight set of pixel dimensions;

[0043] A second convolution submodule, used to convolve the third weight set through a second set number of second convolution kernels to obtain a fourth weight set of pixel dimension and a fourth weight set of channel dimension;

[0044] A weighting submodule is used to perform weighted calculation on the fourth weight set, the second weight set and the feature map that needs to be convolved by the at least one group of convolutional layers to obtain a double-weighted feature map.

[0045] In yet another embodiment of the present disclosure, the first convolutional neural network is a pre-trained VGG-16 network;

[0046] And / or, the first convolution kernel is a 1*1 convolution kernel, and the second convolution kernel is a 3*3 convolution kernel.

[0047] In yet another embodiment of the present disclosure, the first convolution submodule includes:

[0048] A third convolution submodule, configured to convolve the first weight set through a first set number of first convolution kernels to obtain a weight value of at least one channel;

[0049] The parameter optimization submodule is used to set the weight value of the at least one channel that is less than the weight threshold to zero, and retain the weight value of the at least one channel that is not less than the weight threshold, to obtain a second weight set of the channel dimension.

[0050] In yet another embodiment of the present disclosure, the device further includes:

[0051] A third determination module, configured to determine that the image to be identified is an invalid image in response to the quality reference value being not greater than a second set threshold;

[0052] The prompt module is used to output an invalid image prompt message, wherein the invalid image prompt message is used to prompt the user to upload a new image.

[0053] In yet another embodiment of the present disclosure, the reconstruction module is used to perform image super-resolution reconstruction on the reconstructible image using a second convolutional neural network, where the second convolutional neural network is a pre-trained fast super-high resolution convolutional neural network;

[0054] The feature extraction process when the second convolutional neural network performs image super-resolution reconstruction on the reconstructable image includes:

[0055] Using the third convolution kernel to perform channel grouping on the image to be identified;

[0056] The same convolution kernel is used to extract features from the groups obtained by channel grouping to obtain group features;

[0057] Performing mixed calculation on the grouping features to obtain mixed features;

[0058] Feature extraction is performed on the mixed features to obtain an image feature map.

[0059] In yet another embodiment of the present disclosure, the quality reference value of the image to be recognized is determined based on the number of output neurons of the normalization layer of the first convolutional neural network.

[0060] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, the electronic device comprising:

[0061] Memory for storing computer programs;

[0062] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the above-mentioned image processing method is implemented.

[0063] According to another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0064] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer-readable code. When the computer-readable code is executed on a device, a processor in the device executes the above-mentioned image processing method.

[0065] Based on the image processing method, device, electronic device, and storage medium provided by the above embodiments of the present disclosure, a quality reference value of an image to be identified is identified through a pre-trained first convolutional neural network, and the quality reference value is compared with a first set threshold and a second set threshold; when the quality reference value is not less than the second set threshold, the image to be identified is determined to be a valid image; when the quality reference value is greater than the first set threshold and less than the second set threshold, the image to be identified is determined to be a reconstructible image; super-resolution reconstruction is performed on the reconstructible image, and the super-resolution reconstructed image is input into the first convolutional neural network, wherein, since the first convolutional neural network is provided with a dual attention module after at least one pooling layer and before at least one group of convolutional layers, the embodiment of the present disclosure can realize channel weighting and pixel weighting of features before convolution of the convolutional layer, which helps the convolutional layer to perform better convolution processing on the features and improve the recognition accuracy, and can perform super-resolution reconstruction on the image, thereby improving the success rate of image recognition to a certain extent.

[0066] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0068] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0069] Figure 1 A flowchart of an embodiment of the image processing method disclosed herein;

[0070] Figure 2A A flowchart of a dual attention module in the first convolutional neural network of the image processing method disclosed in the present invention;

[0071] Figure 2B A schematic diagram of the operation of a dual attention module of the image processing method disclosed in the present invention;

[0072] Figure 3A A workflow diagram for extracting image features using a second convolutional neural network of the image processing method disclosed herein;

[0073] Figure 3B A schematic diagram of feature extraction of a second convolutional neural network of the image processing method disclosed in the present invention;

[0074] Figure 4 is a schematic structural diagram of an embodiment of an image processing device disclosed in the present invention;

[0075] Figure 5is a structural schematic diagram of another embodiment of the image processing device disclosed in the present invention;

[0076] Figure 6 A structural diagram of an electronic device provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0077] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0078] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0079] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0080] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0081] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0082] The disclosed embodiments can be applied to electronic devices such as computer systems / servers, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with electronic devices such as computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0083] Computer systems / servers and other electronic devices may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0084] SUMMARY OF THE DISCLOSURE

[0085] The technical solution provided by the embodiments of the present disclosure is used in the field of image recognition. When a user needs to upload an ID image to an application and needs the application to recognize the information on the ID image (with the user's full authorization, the application can obtain the user information on the ID image, and the user information may include the user's identification information (such as the user's user name for accessing the application or other identification information that can identify the user. Of course, without the user's full authorization, the user information corresponding to the above ID images belongs to the user's personal privacy information and cannot be obtained by the embodiments of the present disclosure), the image quality judgment and reconstruction based on the visual attention mechanism can be implemented based on the method provided by the embodiments of the present disclosure.

[0086] Exemplary Embodiments

[0087] Figure 1 Flow chart of an embodiment of the image processing method disclosed in the present invention; the image processing method can be applied to an image processing platform (such as a computer system, a server and other electronic devices), such as Figure 1 As shown, the image processing method comprises the following steps:

[0088] In step 101, the image to be recognized is input into a first convolutional neural network to obtain a quality reference value of the image to be recognized. The first convolutional neural network is provided with a dual attention module after at least one pooling layer and before at least one group of convolutional layers.

[0089] In one embodiment, the image to be identified may be an image of a certificate.

[0090] In one embodiment, the first convolutional neural network is a pre-trained VGG-16 network. The VGG-16 network in the prior art is composed of 5 groups of convolutional layers (a total of 13 convolutional layers) and 3 fully connected layers, and each group of convolutional layers is followed by a pooling layer. In order to implement the technical solution provided by the embodiment of the present disclosure, the VGG-16 network in the prior art is used as the basic network for training, and a dual attention module is provided after the pooling layer of each group of convolutional layers, so as to achieve channel weighting and pixel weighting of features before the next group of convolutional layers are convolved, which helps the convolutional layer to perform better convolution processing on the features, thereby realizing image quality discrimination based on the visual attention mechanism. Among them, the dual attention module includes a channel attention mechanism and a pixel attention mechanism, which are used to perform channel weighting and pixel weighting on the features.

[0091] In one embodiment, the process of implementing channel weighting and pixel weighting on features by the pre-trained VGG-16 network through the dual attention module can be found in Figure 2A The embodiment shown is not described in detail here. The number of output neurons of the normalization layer softMax layer after the fully connected layer of the pre-trained VGG-16 network is the quality reference value of the image to be recognized.

[0092] In one embodiment, in the disclosed embodiment, in order to train a pre-trained VGG-16 network, it is necessary to collect images of various types of certificates (identity cards, real estate certificates, and other types of certificates) from existing spare parts, and be able to cover images of different qualities of the same type of certificates, as well as images of different qualities of different types of certificates. The data set must comprehensively cover different situations of different spare parts. Only by training the network model VGG-16 network with the full data set can it be ensured that the trained model has good generalization.

[0093] In step 102, the quality reference value is compared with a first set threshold and a second set threshold.

[0094] In one embodiment, the first set threshold and the second set threshold are two pre-set values ​​used to measure the quality level of the image to be identified, and the quality level may include a valid image, a reconstructible image, and an invalid image. When the quality reference value is not less than the second set threshold, the image to be identified can be determined to be a valid image; when the quality reference value is greater than the first set threshold and less than the second set threshold, the image to be identified is determined to be a reconstructible image; when the quality reference value is not greater than the second set threshold, the image to be identified is determined to be an invalid image, and an invalid image prompt message can be output, and the invalid image prompt message is used to prompt the user to upload a new image.

[0095] Among them, the valid image is an image with very high image quality, which can recognize the information on the image to be recognized; the reconstructible image is an image with slightly lower image quality, which needs to be reconstructed with super resolution before being recognized; the invalid image is an image with very low image quality, which needs to output an invalid image prompt message to remind the user to upload the image again.

[0096] When the quality reference value is not less than the second set threshold, step 103 is executed; when the quality reference value is greater than the first set threshold and less than the second set threshold, step 104 is executed.

[0097] In step 103, when the quality reference value is not less than the second set threshold, the image to be identified is determined to be a valid image.

[0098] In one embodiment, when the image to be identified is determined to be a valid image, information identification can be performed directly on the image.

[0099] In step 104, when the quality reference value is greater than the first set threshold and less than the second set threshold, the image to be identified is determined to be a reconstructible image.

[0100] In step 105, super-resolution reconstruction is performed on the reconstructable image, and the super-resolution reconstructed image is input into the first convolutional neural network, wherein the number of times the super-resolution reconstruction is performed on the reconstructable image does not exceed a set number.

[0101] In one embodiment, the set number of times is the maximum number of times a super-resolution reconstruction is performed on an image to be identified. It is generally not recommended to set the value of the set number of times too large. For example, the set number of times may be 1 or 2.

[0102] In one embodiment, a fast super-resolution reconstruction network may be used to reconstruct the reconstructable image. For the specific reconstruction process, see Figure 3A The illustrated embodiment will not be described in detail here.

[0103] In one embodiment, after super-resolution reconstruction is performed on the reconstructible image, the reconstructed image may be input into the first convolutional neural network again to determine the image quality.

[0104] In the above steps 101 to 105, a quality reference value of the image to be identified is identified through a pre-trained first convolutional neural network, and the quality reference value is compared with a first set threshold and a second set threshold; when the quality reference value is not less than the second set threshold, the image to be identified is determined to be a valid image; when the quality reference value is greater than the first set threshold and less than the second set threshold, the image to be identified is determined to be a reconstructible image; super-resolution reconstruction is performed on the reconstructible image, and the super-resolution reconstructed image is input into the first convolutional neural network, wherein, since a dual attention module for performing channel weighting and pixel weighting on a feature map obtained by convolution of at least one group of convolutional layers is provided after at least one pooling layer of the first convolutional neural network and before at least one group of convolutional layers, the embodiment of the present disclosure can realize image quality recognition based on a visual attention mechanism, improve recognition accuracy, and can improve the success rate of image recognition to a certain extent through super-resolution reconstruction when the image quality is a bit low.

[0105] In order to better illustrate the image processing solution of the present application, another embodiment is described below.

[0106] Figure 2A is a workflow diagram of a dual attention module in the first convolutional neural network of the image processing method disclosed in the present invention, Figure 2B Schematic diagram of the operation of a dual attention module of the image processing method disclosed in the present invention; this embodiment takes the pre-trained VGG-16 network as an example to illustrate how to perform dual attention weighting on feature maps that need to be convolved in at least one group of convolutional layers, such as Figure 2A As shown, the following steps are included:

[0107] In step 201, global pooling is performed on feature maps that need to be convolved in at least one group of convolutional layers to obtain a first weight set in the channel dimension.

[0108] In one embodiment, the existing VGG-16 network includes 5 groups of convolutional layers, and the pre-trained first convolutional neural network is provided with a dual attention module after at least one pooling layer and before at least one group of convolutional layers, so as to realize image quality judgment based on the visual attention mechanism, and the feature map that needs to be convolved for at least one group of convolutional layers can be processed by the dual attention module to obtain a dual-weighted feature map.

[0109] In one embodiment, channel weighting may be performed on feature maps that need to be convolved in at least one group of convolutional layers through steps 201 to 202.

[0110] In one embodiment, after globally pooling the feature maps that need to be convolved in at least one group of convolutional layers, a first weight set of the channel dimension, that is, a preliminary weight set of the channel dimension, can be obtained.

[0111] In step 202, the first weight set is convolved with a first set number of first convolution kernels to obtain a second weight set of channel dimension.

[0112] In one embodiment, the first convolution kernel is a 1*1 convolution kernel.

[0113] In one embodiment, the first weight set is learned through a first set number of 1*1 convolution kernels to obtain a second weight set of channel dimension, which specifically includes: learning the first weight set through a first set number of 1*1 convolution kernels to obtain a weight value of at least one channel; setting the weight value of at least one channel that is less than a weight threshold to zero, and retaining the weight value of at least one channel that is not less than the weight threshold, to obtain a second weight set of channel dimension.

[0114] In one embodiment, the first set number is a pre-set number of times the first weight set is convolved. For example, the first set number may be 2, that is, the first weight set is convolved through two 1*1 convolution layers to obtain the second weight set, that is, the final weight set of the channel dimension.

[0115] In one embodiment, when weighting channels, in order to reduce the complexity and amount of calculation of channel weighting, a weight threshold can be set in advance, and the weight value of at least one channel that is less than the weight threshold is set to zero, and the weight value of at least one channel that is not less than the weight threshold is retained.

[0116] After the second weight set of the channel dimension is calculated through step 202, step 205 can be executed.

[0117] In step 203, for at least one set of feature maps that need to be convolved by the convolution layer, the weight of at least one pixel is calculated through the average pooling layer to obtain a third weight set of pixel dimensions.

[0118] In one embodiment, pixel weighting may be performed on the feature maps of at least one group of convolutional layers through steps 203 and 204.

[0119] In one embodiment, the weight of each pixel in the feature map that needs to be convolved by at least one group of convolutional layers is calculated through an average pooling layer, so as to obtain a third weight set of pixel dimension, that is, a preliminary weight set of pixel dimension.

[0120] In step 204, the third weight set is convolved with a second set number of second convolution kernels to obtain a fourth weight set in pixel dimension.

[0121] In one embodiment, the second convolution kernel is a 3*3 convolution kernel.

[0122] In one embodiment, the second set number is a pre-set number of times the third weight set is learned. For example, the second set number may be 2, that is, after convolving the third weight set with two 3*3 convolution kernels, the fourth weight set, that is, the final weight set in the pixel dimension, can be obtained.

[0123] In step 205, weighted calculation is performed on the fourth weight set, the second weight set, and the feature map that needs to be convolved by at least one group of convolutional layers to obtain a double-weighted feature map.

[0124] In one embodiment, a double-weighted feature map can be obtained by performing weighted calculation on the fourth weight set, the second weight set, and the feature map that needs to be convolved by at least one group of convolutional layers.

[0125] In one embodiment, see Figure 2B , through step 201 and step 202, a second weight set of channel dimension can be obtained, and through step 203 and step 204, a fourth weight set of pixel dimension can be obtained, and weighted calculation is performed on the second weight set and the fourth weight set with at least one set of feature maps that need to be convolved by the convolution layer, so as to obtain a double-weighted feature map.

[0126] Through the above steps 201 to 205, in this embodiment, through the first convolutional neural network with two weighting methods (channel weighting and pixel weighting), image quality recognition based on the visual attention mechanism can be achieved to improve the recognition accuracy; and by setting the weight threshold, the network complexity can be reduced and the classification speed can be improved.

[0127] Figure 3A The following is a workflow diagram of extracting image features by the second convolutional neural network of the image processing method disclosed in the present invention. Figure 3B FIG. 1 is a schematic diagram of feature extraction of a second convolutional neural network of the image processing method disclosed in the present invention; this embodiment takes the feature extraction process when re-resolution reconstruction of a reconstructible image is performed using a Fast Super Resolution Convolution Neural Network (FSRCNN) as an example for illustrative explanation, as shown in FIG. Figure 3A As shown, the following steps are included:

[0128] In step 301, a third convolution kernel is used to perform channel grouping on the reconstructible image.

[0129] In some embodiments of the present application, the third convolution kernel is a 1*1 convolution kernel.

[0130] In one embodiment, the image processing process of the existing FSRCNN network is divided into five parts: feature extraction, compression, nonlinear mapping, expansion, and transposed convolution to obtain a super-resolution image.

[0131] In step 302, the same convolution kernel is used to perform feature extraction on the groups obtained by channel grouping to obtain group features.

[0132] In step 303, the grouping features are mixed and calculated to obtain mixed features.

[0133] In step 304, feature extraction is performed on the mixed features to obtain an image feature map.

[0134] In step 301 to step 304, in order to make the FSRCNN network model lighter, the feature extraction process is finally performed by Figure 3B This can be achieved by using the schematic diagram shown. Figure 3B In the method, a 1*1 convolution kernel is first used to group the channels, and then the same convolution kernel is used to convolve each group obtained by the grouping to obtain the grouping features of each group. Then, the grouping features in each channel group are mixed and convolution is performed again, so that the feature extraction of the image to be recognized can be realized.

[0135] In one embodiment, after completing feature extraction, the FSRCNN network can obtain a super-resolution image through operations such as nonlinear mapping, expansion, and transposed convolution.

[0136] Through the above steps 301 to 304, in this embodiment, the convolutional layer is replaced by Figure 3B The module can achieve the purpose of reducing parameters, and by correlating the features between channel groups, it can achieve better learning ability and reconstruct a clearer image.

[0137] Corresponding to the above-mentioned embodiments of the image processing method, the present disclosure also provides corresponding embodiments of the image processing device.

[0138] Figure 4 FIG. 1 is a schematic diagram of a structure of an embodiment of an image processing device disclosed in the present invention. The device is applied on an image processing platform, such as Figure 4 As shown, the device comprises:

[0139] A quality determination module 41, used for inputting the image to be identified into a first convolutional neural network to obtain a quality reference value of the image to be identified, wherein the first convolutional neural network is provided with a dual attention module after at least one pooling layer and before at least one group of convolutional layers, and the dual attention module is used for performing channel weighting and pixel weighting on a feature map to be convolved by at least one group of convolutional layers;

[0140] A comparison module 42, configured to compare the quality reference value with a first set threshold and a second set threshold;

[0141] A first determination module 43, configured to determine that the image to be identified is a valid image when the quality reference value is not less than a second set threshold;

[0142] A second determination module 44, configured to determine that the image to be identified is a reconstructible image when the quality reference value is greater than a first set threshold and less than a second set threshold;

[0143] The reconstruction module 45 is used to perform super-resolution reconstruction on the reconstructable image and input the super-resolution reconstructed image into the first convolutional neural network, wherein the number of times the super-resolution reconstruction is performed on the reconstructable image does not exceed a set number.

[0144] Figure 5 FIG. 1 is a structural diagram of another embodiment of the image processing device disclosed in the present invention. Figure 5 As shown, in Figure 4 Based on the illustrated embodiment, in one embodiment, the quality determination module 41 includes:

[0145] A global pooling submodule 411 is used to perform global pooling on at least one set of feature maps that need to be convolved in the convolution layer to obtain a first weight set of the channel dimension;

[0146] A first convolution submodule 412, configured to convolve the first weight set through a first set number of first convolution kernels to obtain a second weight set of channel dimension;

[0147] The average pooling submodule 413 is used to calculate the weight of at least one pixel of the feature map that needs to be convolved by the at least one group of convolution layers through the average pooling layer to obtain a third weight set in the pixel dimension, and obtain a third weight set in the pixel dimension;

[0148] A second convolution submodule 414 is used to convolve the third weight set through a second set number of second convolution kernels to obtain a fourth weight set in pixel dimension and a fourth weight set in channel dimension;

[0149] The weighting submodule 415 is used to perform weighted calculation on the fourth weight set, the second weight set and the feature map that needs to be convolved by at least one group of convolution layers to obtain a double-weighted feature map.

[0150] In one embodiment, the first convolution submodule 412 includes:

[0151] A third convolution submodule 4121, configured to convolve the first weight set through a first set number of first convolution kernels to obtain a weight value of at least one channel;

[0152] The parameter optimization submodule 4122 is used to set the weight value of at least one channel that is less than the weight threshold to zero, and retain the weight value of at least one channel that is not less than the weight threshold, to obtain a second weight set of the channel dimension.

[0153] In one embodiment, the apparatus further comprises:

[0154] A third determination module 46 is configured to determine that the image to be identified is an invalid image in response to the quality reference value being not greater than a second set threshold;

[0155] The prompt module 47 is used to output an invalid image prompt message, and the invalid image prompt message is used to prompt the user to upload a new image.

[0156] In one embodiment, the reconstruction module 45 is used to perform image super-resolution reconstruction on the reconstructable image using a second convolutional neural network, where the second convolutional neural network is a pre-trained fast super-resolution convolutional neural network;

[0157] Among them, the feature extraction process of the second convolutional neural network when performing image super-resolution reconstruction on the reconstructible image includes:

[0158] The third convolution kernel is used to group the channels of the reconstructible image;

[0159] The same convolution kernel is used to extract features from the groups obtained by channel grouping to obtain group features;

[0160] Perform mixed calculation on the grouping features to obtain mixed features;

[0161] Feature extraction is performed on the mixed features to obtain an image feature map.

[0162] In one embodiment, the quality reference value of the image to be recognized is determined based on the number of output neurons of the normalization layer of the first convolutional neural network.

[0163] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0164] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art may understand and implement it without creative work.

[0165] Below, reference Figure 6 An electronic device according to an embodiment of the present disclosure is described, in which a device for implementing the method according to an embodiment of the present disclosure may be integrated. Figure 6A structural diagram of an electronic device provided in an exemplary embodiment of the present disclosure, such as Figure 6 As shown, the electronic device 6 includes one or more processors 61, one or more computer-readable storage media memory 62, and a computer program stored in the memory and executable on the processor. When the program in the memory 62 is executed, the above-mentioned image processing method can be implemented.

[0166] Specifically, in practical applications, the electronic device may further include components such as an input device 63 and an output device 64, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will appreciate that Figure 6 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or certain components, or different component arrangements.

[0167] The processor 61 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0168] The memory 62 may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products may be stored on a computer-readable storage medium, and the processor 61 may run the computer program products to implement the sound source localization method of each embodiment of the present disclosure and / or other desired functions.

[0169] In one example, the electronic device may further include: an input device 63 and an output device 64, these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0170] In addition, the input device 63 may include, for example, a keyboard, a mouse, and the like.

[0171] The output device 64 can output various information to the outside, including the determined distance information, direction information, etc. The output device 64 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0172] Of course, to simplify, Figure 6Only some of the components related to the present disclosure in the electronic device 6 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 6 may also include any other appropriate components.

[0173] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the sound source localization method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0174] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0175] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the sound source localization method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0176] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0177] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0178] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0179] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0180] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0181] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting the image to be recognized into a first convolutional neural network to obtain a quality reference value of the image to be recognized, wherein the first convolutional neural network is provided with a dual attention module after each pooling layer in at least one pooling layer and before each convolution layer in at least one group of convolutional layers, and the dual attention module is used to perform channel weighting and pixel weighting on a feature map to be convolved in the at least one group of convolutional layers; comparing the quality reference value with a first set threshold and a second set threshold; When the quality reference value is not less than a second set threshold, determining that the image to be identified is a valid image; When the quality reference value is greater than a first set threshold and less than the second set threshold, determining that the image to be identified is a reconstructible image; Performing super-resolution reconstruction on the reconstructable image, and inputting the super-resolution reconstructed image into the first convolutional neural network, wherein the number of times the super-resolution reconstruction is performed on the image to be identified does not exceed a set number; The process of performing channel weighting on the feature map to be convolved by the at least one group of convolutional layers by the dual attention module includes: Performing global pooling on the feature maps that need to be convolved in the at least one group of convolutional layers to obtain a first weight set of channel dimensions; Convolving the first weight set with a first set number of first convolution kernels to obtain a second weight set of channel dimension, including: convolving the first weight set with a first set number of first convolution kernels to obtain a weight value of at least one channel; setting the weight value of the at least one channel that is less than a weight threshold to zero, and retaining the weight value of the at least one channel that is not less than the weight threshold to obtain the second weight set of the channel dimension.

2. The method according to claim 1, characterized in that The process of the dual attention module processing the feature map that needs to be convolved by the at least one group of convolutional layers also includes: For the feature graphs to be convolved by the at least one group of convolutional layers, a weight of at least one pixel is calculated through an average pooling layer to obtain a third weight set of pixel dimensions; Convolving the third weight set with a second set number of second convolution kernels to obtain a fourth weight set in pixel dimension; The fourth weight set, the second weight set, and the feature map that needs to be convolved by the at least one group of convolutional layers are weighted to obtain a double-weighted feature map.

3. The method according to claim 2, characterized in that The first convolutional neural network is a pre-trained VGG-16 network; And / or, the first convolution kernel is a 1*1 convolution kernel, and the second convolution kernel is a 3*3 convolution kernel.

4. The method according to claim 1, characterized in that: The method further comprises: In response to the quality reference value being not greater than a second set threshold, determining that the image to be identified is an invalid image; Outputting an invalid image prompt message, wherein the invalid image prompt message is used to prompt the user to upload a new image.

5. The method according to claim 1, characterized in that The super-resolution reconstruction of the reconstructable image comprises: Performing image super-resolution reconstruction on the reconstructable image using a second convolutional neural network, where the second convolutional neural network is a pre-trained fast super-high resolution convolutional neural network; The feature extraction process when the second convolutional neural network performs image super-resolution reconstruction on the reconstructable image includes: Using a third convolution kernel to perform channel grouping on the image to be identified; The same convolution kernel is used to extract features from the groups obtained by channel grouping to obtain group features; Performing mixed calculation on the grouping features to obtain mixed features; Feature extraction is performed on the mixed features to obtain an image feature map.

6. The method according to claim 1, characterized in that The quality reference value of the image to be recognized is determined based on the number of output neurons of the normalization layer of the first convolutional neural network.

7. An electronic device, characterized in that: include: Memory, used to store computer products; A processor is used to execute the computer product stored in the memory, and when the computer product is executed, the method described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images