A deep learning-based image recognition method and related device

Abstract

The present application discloses an image recognition method based on deep learning and a related device. The method obtains multiple feature maps to be processed corresponding to the image to be recognized through feature extraction. Based on a preset first expansion factor, multiple rounds of first convolution operations are performed on each feature map to be processed, so as to determine the first total feature map corresponding to each image to be processed according to the feature recognition results obtained in each round of the first convolution operation. Then, the recognition result of the image to be recognized is determined by a decoder. Since the first expansion factor corresponding to each round of the first convolution operation is different, the feature recognition results obtained in each round of the first convolution operation are in different receptive fields. And the feature recognition results of each round of the first convolution operation will be used as the input item of the next round, so the last round of the first convolution operation can obtain the first total feature map that represents the fusion of feature results under different receptive fields. The above process can greatly reduce the number of parameters of the neural network model and improve the model convergence speed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image recognition method based on deep learning and related devices. Background Art

[0002] Image recognition technology is mostly based on deep learning image classification algorithms, which use pooling layers or multi-layer convolution to extract the features of the image to be recognized. In order to improve classification accuracy, related technologies often use downsampling operations to expand the receptive field range, thereby increasing the semantic information of the feature image. However, expanding the receptive field range will lead to the loss of feature image detail information.

[0003] To solve the above problems, feature pyramids are currently used to extract features of the image to be identified in different receptive fields through multiple convolution kernels, and then fuse the features in different receptive fields. Although this processing method can repair the detailed information of the feature map to a certain extent, it greatly increases the number of parameters of the neural network model, resulting in a decrease in the convergence speed of the neural network. Summary of the invention

[0004] The embodiment of the present application provides an image recognition method based on deep learning and a related device, which is used to perform multiple rounds of convolution on the feature map to be processed through expansion factors of different values ​​to fuse the feature recognition results under different receptive fields, reduce the number of parameters of the neural network model, and improve the convergence speed.

[0005] In a first aspect, an embodiment of the present application provides an image recognition method based on deep learning, the method comprising:

[0006] Perform feature extraction on the image to be identified, and obtain a plurality of feature maps to be processed representing different dimensions of the image to be identified; wherein the sizes of the feature maps to be processed are the same;

[0007] Based on a preset first dilation factor, multiple rounds of first convolution operations are performed on each feature map to be processed, so as to determine a first total feature map corresponding to each feature map to be processed according to a feature recognition result obtained by each round of the first convolution operation; wherein the first dilation factor corresponding to each round of the first convolution operation is different, and different values ​​of the first dilation factor represent different receptive fields corresponding to the feature recognition result;

[0008] The preset category to which the first total feature map belongs is determined by a decoder, and the preset category is used as the recognition result of the image to be recognized; wherein the first convolution operation process is as follows:

[0009] Based on the first expansion factor corresponding to this round, a first convolution kernel is used to perform a convolution operation on the input item to obtain a first sub-feature map corresponding to the input item; and a second convolution kernel is used to perform a convolution operation on the first sub-feature map to obtain a feature recognition result of the input item under the receptive field corresponding to the first expansion factor; wherein, the input item of the first round of the first convolution operation is the feature map to be processed, and the input item of the non-first round of the first convolution operation is the feature recognition result obtained in the previous round; the sizes of the first convolution kernel and the second convolution kernel are different.

[0010] The embodiment of the present application extracts features from the image to be identified, and obtains multiple feature maps to be processed corresponding to the image to be identified. Based on the pre-set first expansion factor, multiple rounds of first convolution operations are performed on each feature map to be processed, so as to determine the first total feature map corresponding to each image to be processed according to the feature recognition results obtained in each round of the first convolution operation. The preset classification of the first total feature map is determined by the decoder, and then the recognition result of the image to be identified is obtained. Since each round of the first convolution operation corresponds to a different first expansion factor, the first expansion factor with different values ​​represents that the receptive field corresponding to the feature recognition result is different, so the feature recognition result obtained by each round of the first convolution operation is in a different receptive field. And the feature recognition result of each round of the first convolution operation will be used as the input item of the next round, so the last round of the first convolution operation can obtain the first total feature map representing the fusion of feature results under different receptive fields. Through the above process, the number of parameters of the neural network model can be greatly reduced, and the model convergence speed can be improved.

[0011] In some possible embodiments, the preset dimensions of the first convolution kernel and the second convolution kernel are the same, and after acquiring a plurality of feature maps to be processed representing different dimensions of the image to be recognized, the method further includes:

[0012] Before performing multiple rounds of first convolution operations on each feature map to be processed, dimension processing is performed on each feature map to be processed so that the dimension of each feature map to be processed is the same as the preset dimension.

[0013] In the embodiment of the present application, the preset dimensions of the first convolution kernel and the second convolution kernel are the same. Before performing multiple rounds of the first convolution operation on each feature map to be processed, it is necessary to perform dimension processing on each feature map to be processed so that the dimension of each feature map to be processed is the same as the preset dimension. Therefore, the sub-feature map obtained after the first convolution kernel performs a convolution operation on the feature map to be processed can be directly used for convolution with the second convolution kernel, thereby improving the processing rate.

[0014] In some possible embodiments, the number of the second convolution kernels is greater than the number of the first convolution kernels, and the using the first convolution kernel to perform a convolution operation on the input item includes:

[0015] Performing a convolution operation on each image in the input item to obtain a first sub-feature map corresponding to each image; wherein the image is a feature map to be processed or a feature recognition result;

[0016] The using a second convolution kernel to perform a convolution operation on the first sub-feature map includes:

[0017] A convolution operation is performed on each of the images in the input item, and pixel values ​​at the same position of each of the images are added during the convolution process to obtain feature extraction results corresponding to all the images in the input item.

[0018] In the embodiment of the present application, the number of second convolution kernels is greater than the number of first convolution kernels. When the second convolution kernel is used to convolve the first sub-feature map output by the first convolution kernel, multiple feature recognition results of the first sub-feature map in different dimensions can be obtained, thereby improving the detail information of the image.

[0019] In some possible embodiments, determining, by a decoder, a preset category to which the first total feature map belongs includes:

[0020] Controlling the regression head to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map; and,

[0021] Controlling the classification head to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map; wherein the second expansion factor has a different value from the first expansion factor;

[0022] If the classification detection result indicates that the first total feature map is a recognizable image, category recognition is performed on the region of interest of the first total feature map, and the preset category to which the first total feature map belongs is determined based on the recognition result; if it is not the recognizable image, a prompt message indicating that the image cannot be recognized is output.

[0023] The embodiment of the present application controls the regression head to determine the region of interest of the first total feature map, and controls the classification head to determine the classification detection result of the first total feature map. When the classification detection result indicates that the first total feature map is a recognizable image, the region of interest is identified by category to determine the preset category to which it belongs. The above-mentioned regression head and classification head are processed synchronously in parallel to improve the processing rate.

[0024] In some possible embodiments, the controlling regression head performs a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map, including:

[0025] Based on the second expansion factor, using a third convolution kernel to perform a convolution operation on the first total feature map to obtain a second sub-feature map under a receptive field corresponding to the second expansion factor; wherein the third convolution kernel has the same size as the first convolution kernel;

[0026] A fourth convolution kernel is used to perform a convolution operation on the second sub-feature map to obtain a second total feature map corresponding to the second sub-feature map; and a fifth convolution kernel is used to perform a convolution operation on the second total feature map to obtain the region of interest; wherein the fourth convolution kernel has the same size as the second convolution kernel, and the number of the fourth convolution kernels is greater than the number of the third convolution kernels; and the size of the fifth convolution kernel is the same as the first convolution kernel.

[0027] In the embodiment of the present application, the regression head is controlled to perform a convolution operation on the first total feature map based on the second expansion factor, and a third convolution kernel is used to perform a convolution operation on the first total feature map to obtain a second sub-feature map under the receptive field corresponding to the second expansion factor. Then, a fourth convolution kernel is used to perform a convolution operation on the second sub-feature map to obtain a second total feature map representing the feature fusion result of the second sub-feature map. Finally, a fifth convolution kernel with the ability to recognize the region of interest is used to determine the region of interest from the second total feature map to improve the accuracy of selecting the region of interest.

[0028] In some possible embodiments, the controlling classification head performs a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map, including:

[0029] Based on the second expansion factor, using a sixth convolution kernel to perform a convolution operation on the first total feature map to obtain the third sub-feature map under the receptive field corresponding to the second expansion factor; wherein the sixth convolution kernel has the same size as the first convolution kernel;

[0030] A seventh convolution kernel is used to perform a convolution operation on the third sub-feature map to obtain a third total feature map corresponding to the third sub-feature map; and an eighth convolution kernel is used to perform a convolution operation on the third total feature map to obtain the classification detection result; wherein, the sizes of the sixth convolution kernel and the eighth convolution kernel are the same as those of the first convolution kernel; the seventh convolution kernel is the same as the size of the second convolution kernel, and the number of the seventh convolution kernels is greater than the number of the sixth convolution kernels.

[0031] The embodiment of the present application controls the classification head to perform a convolution operation on the first total feature map based on the second expansion factor, using the sixth convolution kernel to obtain a third sub-feature map under the receptive field corresponding to the second expansion factor, and then uses the seventh convolution kernel to perform a convolution operation on the third sub-feature map to obtain a third total feature map representing the feature fusion result of the third sub-feature map. Finally, the eighth convolution kernel with classification and detection capabilities is used to screen out the third total feature map that can be recognized by the neural network model to improve the model recognition accuracy.

[0032] In some possible embodiments, the classification head and the regression head in the decoder each include at least one dilated convolution block.

[0033] The present application performs multiple rounds of convolution on the feature map to be processed through expansion factors of different values ​​to fuse the feature recognition results under different receptive fields. There is no need to perform multi-scale feature fusion similar to the feature pyramid, which avoids the feature loss caused by the different feature weights in the feature fusion process. The decoder of the present application only requires one regression head and one classification head to ensure the recognition accuracy of the classification results.

[0034] In a second aspect, an embodiment of the present application provides an image recognition device based on deep learning, the device comprising:

[0035] A feature extraction module is configured to perform feature extraction on the image to be identified, and obtain a plurality of feature maps to be processed representing different dimensions of the image to be identified; wherein the sizes of the feature maps to be processed are the same;

[0036] The feature fusion module is configured to perform multiple rounds of first convolution operations on each feature map to be processed based on a preset first dilation factor, so as to determine a first total feature map corresponding to each feature map to be processed according to a feature recognition result obtained by each round of the first convolution operation; wherein the first dilation factor corresponding to each round of the first convolution operation is different, and different values ​​of the first dilation factor represent different receptive fields corresponding to the feature recognition result;

[0037] The image recognition module is configured to determine the preset category to which the first total feature map belongs through a decoder, and use the preset category as the recognition result of the image to be recognized; wherein the first convolution operation process is as follows:

[0038] Based on the first expansion factor corresponding to this round, a first convolution kernel is used to perform a convolution operation on the input item to obtain a first sub-feature map corresponding to the input item; and a second convolution kernel is used to perform a convolution operation on the first sub-feature map to obtain a feature recognition result of the input item under the receptive field corresponding to the first expansion factor; wherein, the input item of the first round of the first convolution operation is the feature map to be processed, and the input item of the non-first round of the first convolution operation is the feature recognition result obtained in the previous round; the sizes of the first convolution kernel and the second convolution kernel are different.

[0039] In some possible embodiments, the preset dimensions of the first convolution kernel and the second convolution kernel are the same, and after performing the step of acquiring a plurality of feature maps to be processed representing different dimensions of the image to be recognized, the feature extraction module is further configured as follows:

[0040] Before performing multiple rounds of first convolution operations on each feature map to be processed, dimension processing is performed on each feature map to be processed so that the dimension of each feature map to be processed is the same as the preset dimension.

[0041] In some possible embodiments, the number of the second convolution kernels is greater than the number of the first convolution kernels, and the convolution operation on the input item using the first convolution kernel is performed, and the feature fusion module is configured as follows:

[0042] Performing a convolution operation on each image in the input item to obtain a first sub-feature map corresponding to each image; wherein the image is a feature map to be processed or a feature recognition result;

[0043] The convolution operation on the first sub-feature map is performed using the second convolution kernel, and the feature fusion module is configured as follows:

[0044] A convolution operation is performed on each of the images in the input item, and pixel values ​​at the same position of each of the images are added during the convolution process to obtain feature extraction results corresponding to all the images in the input item.

[0045] In some possible embodiments, the determining, by a decoder, of a preset category to which the first total feature map belongs is performed, and the image recognition module is configured as follows:

[0046] Controlling the regression head to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map; and,

[0047] Controlling the classification head to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map; wherein the second expansion factor has a different value from the first expansion factor;

[0048] If the classification detection result indicates that the first total feature map is a recognizable image, category recognition is performed on the region of interest of the first total feature map, and the preset category to which the first total feature map belongs is determined according to the recognition result.

[0049] In some possible embodiments, the control regression head is executed to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map, and the image recognition module is configured as follows:

[0050] Based on the second expansion factor, using a third convolution kernel to perform a convolution operation on the first total feature map to obtain a second sub-feature map under a receptive field corresponding to the second expansion factor; wherein the third convolution kernel has the same size as the first convolution kernel;

[0051] A fourth convolution kernel is used to perform a convolution operation on the second sub-feature map to obtain a second total feature map corresponding to the second sub-feature map; and a fifth convolution kernel is used to perform a convolution operation on the second total feature map to obtain the region of interest; wherein the sizes of the third convolution kernel and the fifth convolution kernel are the same as those of the first convolution kernel; the fourth convolution kernel is the same as the size of the second convolution kernel, and the number of the fourth convolution kernels is greater than the number of the third convolution kernels.

[0052] In some possible embodiments, the control classification head is executed to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map, and the image recognition module is configured as follows:

[0053] Based on the second expansion factor, using a sixth convolution kernel to perform a convolution operation on the first total feature map to obtain the third sub-feature map under the receptive field corresponding to the second expansion factor; wherein the sixth convolution kernel has the same size as the first convolution kernel;

[0054] A seventh convolution kernel is used to perform a convolution operation on the third sub-feature map to obtain a third total feature map corresponding to the third sub-feature map; and an eighth convolution kernel is used to perform a convolution operation on the third total feature map to obtain the classification detection result; wherein, the sizes of the sixth convolution kernel and the eighth convolution kernel are the same as those of the first convolution kernel; the seventh convolution kernel is the same as the size of the second convolution kernel, and the number of the seventh convolution kernels is greater than the number of the sixth convolution kernels.

[0055] In some possible embodiments, the classification head and the regression head in the decoder each include at least one dilated convolution block.

[0056] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0057] processor;

[0058] a memory for storing instructions executable by the processor;

[0059] The processor is configured to execute the instructions to implement any one of the methods provided in the first aspect of the present application.

[0060] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods provided in the first aspect of the present application.

[0061] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any method provided in the first aspect of the present application.

[0062] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A schematic diagram of feature pyramid FPN is shown for the embodiment of the present application;

[0065] Figure 2 A schematic diagram of the Yolof network structure shown in an embodiment of the present application;

[0066] Figure 3a The overall flow chart of the image recognition method based on deep learning shown in the embodiment of the present application;

[0067] Figure 3b A schematic diagram of the neural network structure of the present application is shown in the embodiment of the present application;

[0068] Figure 4 This is a structural diagram of an image recognition device 400 based on deep learning according to an embodiment of the present application;

[0069] Figure 5 This is a schematic diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " will mean or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0071] In the description of the embodiments of the present application, unless otherwise specified, the term "multiple" refers to two or more, and other quantifiers are similar and should be understood. The preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features therein may be combined with each other if there is no conflict.

[0072] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0073] As mentioned above, in order to improve classification accuracy, related technologies often use downsampling operations to expand the receptive field range, thereby increasing the semantic information of the feature image. However, expanding the receptive field range will lead to the loss of feature image detail information.

[0074] In related technologies, feature pyramids are often used to extract features of the image to be identified in different receptive fields through multiple convolution kernels, and then feature fusion is performed on the features in different receptive fields. Specifically, Figure 1 The network structure of feature pyramid FPN is shown. Figure 1 As shown, the FPN network contains a bottom-up convolution circuit (such as Figure 1 As shown on the left), that is, a top-down feature convolution. And a top-down convolution circuit (as shown Figure 1As shown on the right). Since the high-level features have more semantic features, and the low-level features have fewer semantic features but more position information, FPN fuses the feature images adjacent to the feature map on the left to connect the low-resolution, high-semantic information high-level features with the high-resolution, low-semantic information low-level features from top to bottom, so that the features at all scales have rich semantic information to improve the detail information of the feature image. The specific method is to upsample the higher-level features of the two feature layers by 2 times (that is, insert new elements between pixels based on the pixels of the original image using a suitable interpolation algorithm to double the image size). The lower-level features change the number of channels of the lower-level features through 1×1 convolution, and then simply add the corresponding elements of the results of upsampling and 1×1 convolution.

[0075] The advantage of this method is that it can extract features from images of each scale, generate multi-scale feature images, and all feature images have strong semantic information. The disadvantage of this method is that the model has a large number of parameters and occupies a large amount of memory.

[0076] In order to solve the above problems, the inventive concept of the present application is as follows: by extracting features from the image to be identified, a plurality of feature maps to be processed corresponding to the image to be identified are obtained. Based on the preset first expansion factor, multiple rounds of first convolution operations are performed on each feature map to be processed, so as to determine the first total feature map corresponding to each image to be processed according to the feature recognition results obtained by each round of the first convolution operation. The preset classification of the first total feature map is determined by the decoder, and then the recognition result of the image to be identified is obtained. Since the first expansion factor corresponding to each round of the first convolution operation is different, the first expansion factor with different values ​​represents that the receptive field corresponding to the feature recognition result is different, so the feature recognition result obtained by each round of the first convolution operation is in a different receptive field. And the feature recognition result of each round of the first convolution operation will be used as the input item of the next round, so the last round of the first convolution operation can obtain the first total feature map representing the fusion of feature results under different receptive fields. Through the above process, the number of parameters of the neural network model can be greatly reduced, and the model convergence speed can be improved.

[0077] In order to facilitate the understanding of the technical solution provided by this application, the Yolof neural network is briefly described first. Figure 2 As shown:

[0078] Figure 2 The network structure of the Yolof neural network is shown, including a feature extraction skeleton Backbone, an encoder Encoder and a decoder Decoder. The Backbone is used to extract features from the image to be identified to obtain the feature image of the image to be identified.

[0079] The output of Backbone in this network structure includes two versions, one of which is a feature image with a sampling rate of 32 and a channel number of 2048 relative to the input image, that is, Figure 2 The second is the feature image with a sampling rate of 16 and a channel number of 2048 relative to the input image, that is, Figure 2 DC5 shown.

[0080] Next, the feature image is input into the Encoder. The network structure uses the dilated encoder DilatedEncoder as the Encoder. The Dilated Encoder takes the features output by Backbone as input, first uses 1x1 convolution and 3x3 convolution to reduce the number of feature channels to 512, and then uses 4 consecutive residual modules to extract features. In each residual module, first use 1x1 convolution to reduce the number of feature channels to 1 / 4 of the original, then use 3x3 dilated convolution to increase the receptive field, and finally use 1x1 convolution to expand the number of feature channels by 4 times. The dilation factors of the dilated convolution in the 4 residual modules are 2, 4, 6, and 8, respectively. In this way, after 4 rounds of dilated convolution operations (one round for each dilation factor), the feature fusion results of the feature image under different receptive fields can be obtained.

[0081] Finally, the feature fusion result is input into the Decoder. In Yolof's Decoder, the regression head (Conv-BN-ReLU) contains 4 dilated convolution blocks, and the classification head (Conv-BN-ReLU) contains 2 dilated convolution blocks. The number of dilated convolution blocks represents how many times the dilated convolution operation needs to be performed. Each anchor in the regression head has a prediction box objectness prediction (i.e., region of interest), and the final classification score is obtained by multiplying the output of the classification branch by the objectness prediction. That is, the regression head is used to determine the region of interest in the feature fusion result, and the classification head is used to classify and identify the region of interest of the feature fusion result to obtain the category to which the input image belongs.

[0082] The feature fusion process of the encoder in the Yolof neural network only uses the first-level feature for detection, without the need for Figure 1 The feature pyramid FPN shown requires multiple layers of feature detection. Compared with FPN, the memory usage is greatly reduced. Based on this, the present application is based on the network structure of Yolof, and uses convolution kernels of different sizes to perform multiple rounds of convolution operations on the feature map to be processed to obtain the first total feature map that represents the fusion of feature results under different receptive fields. The number of parameters required for the operation is greatly reduced compared to Yolof, which improves the model convergence rate.

[0083] The following is a detailed description of an image recognition method based on deep learning provided by an embodiment of the present application in conjunction with the accompanying drawings. Figure 3a As shown, the following steps are included:

[0084] Step 301: extracting features from the image to be identified, and obtaining a plurality of feature maps to be processed representing different dimensions of the image to be identified; wherein the sizes of the feature maps to be processed are the same;

[0085] The embodiment of the present application selects RegNetX-400MF as the basis for constructing a feature extraction network, which includes 4 modules. The feature map output by each module is 2 times smaller than the feature map output by the previous module, and the downsampling is completed by increasing the convolution step size. Each module is composed of a different number of blocks cascaded, the numbers are [1,2,7,12], forming a residual network structure. Each convolution layer is followed by a batch normalization layer (Batch Normalization) and a corrected nonlinear function layer (ReLU). The Batch Normalization layer can normalize the output, and the ReLU layer adds nonlinearity to the output of the upper layer. The feature dimension of RegNetX-400MF is 384. That is, after feature extraction of N images to be identified, feature images of (N, 384, H, W) will be obtained, where H and W are the length and width of the image. That is, N 384-dimensional feature maps with a size of H×W to be processed.

[0086] Step 302: performing multiple rounds of first convolution operations on each feature map to be processed based on a preset first expansion factor, so as to determine a first total feature map corresponding to each feature map to be processed according to a feature recognition result obtained by each round of the first convolution operation; wherein the first expansion factor corresponding to each round of the first convolution operation is different, and different values ​​of the first expansion factor represent different receptive fields corresponding to the feature recognition result;

[0087] The preset category to which the first total feature map belongs is determined by a decoder, and the preset category is used as the recognition result of the image to be recognized; wherein the first convolution operation process is as follows:

[0088] Based on the first expansion factor corresponding to this round, a first convolution kernel is used to perform a convolution operation on the input item to obtain a first sub-feature map corresponding to the input item; and a second convolution kernel is used to perform a convolution operation on the first sub-feature map to obtain a feature recognition result of the input item under the receptive field corresponding to the first expansion factor; wherein, the input item of the first round of the first convolution operation is the feature map to be processed, and the input item of the non-first round of the first convolution operation is the feature recognition result obtained in the previous round; the sizes of the first convolution kernel and the second convolution kernel are different.

[0089] In some possible embodiments, since the dimension of the image to be convolved needs to be the same as the dimension of the convolution kernel, the preset dimensions of the first convolution kernel and the second convolution kernel can be set to be the same, so that the sub-feature map obtained after the first convolution kernel performs a convolution operation on the feature map to be processed can be directly used for convolution with the second convolution kernel, thereby improving the processing rate.

[0090] Furthermore, before performing multiple rounds of first convolution operations on each feature map to be processed, the embodiment of the present application performs dimension processing on each feature map to be processed so that the dimension of each feature map to be processed is the same as the preset dimension.

[0091] Specifically, the feature image extracted by the feature extraction backbone in the Yolof neural network is (N, 512, 25, 25). That is, N 512-dimensional feature images with a size of 25×25. This application uses the dilated convolution architecture of Yolof here, so it requires the input of a feature map to be processed of size (N, 512, 25, 25). Therefore, here we first use 1x1 convolution (512, 1, 1) to increase the dimension of the (N, 384, H, W) feature map obtained in the above step 301, and adjust the size to turn it into a feature map to be processed of size (N, 512, 25, 25), and then perform a standard 3x3 convolution operation on the (N, 512, 25, 25) feature map to be processed obtained above to refine the context.

[0092] In order to obtain the feature fusion results of the feature map to be processed under different receptive fields, the embodiment of the present application sets a plurality of numerical first expansion factors, and each numerical first expansion factor corresponds to a round of first convolution operation. Thus, the first total feature map representing the feature fusion results of the feature map to be processed under different receptive fields can be obtained by performing multiple first convolution operations on the image to be processed.

[0093] In the embodiment of the present application, the number of the second convolution kernels is greater than the number of the first convolution kernels. When performing the first convolution operation of the above step 302, the first convolution kernel is first used to perform a convolution operation on each image in the input item to obtain a first sub-feature map corresponding to each image; wherein the image is a feature map to be processed or a feature recognition result. Then, the second convolution kernel is used to perform a convolution operation on each image in the input item when performing a convolution operation on the first sub-feature map, and the pixel values ​​at the same position of each image are added during the convolution process to obtain feature extraction results corresponding to all images in the input item.

[0094] Specifically, the values ​​of the first expansion factor of the present application are 2, 4, 6 and 8 respectively. That is, 4 rounds of the first convolution operation are required to be performed on the feature map to be processed. When performing the first round of the first convolution operation, the (N, 512, 25, 25) feature map to be processed obtained above is first reduced to the required (N, 128, 25, 25) feature map to be processed using 1x1 convolution, and then the first expansion factor with a value of 2 is used. Through the first convolution kernel of 128 dimensions and size of 3×3, a convolution operation is performed on the feature map to be processed, which will obtain a first sub-feature map of size (N, 128, 25, 25). Taking one feature map to be processed as an example, the first sub-feature maps of the feature map to be processed in 128 different dimensions (i.e., 128 first sub-feature maps) can be obtained through the convolution operation of the first convolution kernel.

[0095] Next, the first sub-feature map (N, 128, 25, 25) is convolved with 128 128-dimensional second convolution kernels of size 1x1, and the pixel values ​​at the same position of different images are added during the convolution process to obtain N×128 feature recognition results, and the size of the feature recognition results is (N, 128, 25, 25). At this point, the first round of the first convolution operation is completed, that is, the first round of the first convolution operation, through the convolution operation of the first convolution kernel and the second convolution kernel, finally obtains the feature fusion result of each first sub-feature map in 128 dimensions under the receptive field corresponding to the first expansion factor of 2.

[0096] Furthermore, the first expansion factors with values ​​of 4, 6 and 8 are used respectively, and the above-mentioned operation of performing convolution operation through the first convolution kernel and the second convolution kernel is repeated. Compared with the first round of the first convolution operation, the input item of the non-first round of the first convolution operation is the feature recognition result obtained in the previous round. It is equivalent to that the input item obtained by the first round of the first convolution operation is the feature recognition result obtained in the previous round. The second round of the first convolution operation uses the feature recognition result 1 obtained in the first round as an input item to obtain the feature recognition result 2 corresponding to the input item. The third round of the first convolution operation uses the feature recognition result 2 as an input item to obtain the feature recognition result 3. The fourth round of the first convolution operation uses the feature recognition result 3 as an input item to obtain the feature recognition result 4. Since the receptive field of each round of feature recognition results is different, the feature recognition result 4 obtained in the fourth round is equivalent to the feature recognition result that integrates different receptive fields, that is, the first total feature map.

[0097] The Yolof decoder processing stage needs to perform convolution operations on the feature image (N, 128, 25, 25) based on the corresponding expansion factor of each round through 128 128-dimensional, 3×3 convolution kernels. The number of parameters required for each round is 128x3x3x128=147456, while in the above process, the number of parameters required for the first convolution operation of each round of this application is 128x3x3+128x128x1x1=17536. Compared with Yolof, the number of parameters per round is reduced by nearly 9 times, which greatly improves the processing speed of the neural network.

[0098] When the embodiment of the present application determines the preset category to which the first total feature map belongs through the decoder, it is necessary to control the regression head to perform a second convolution operation on the first total feature map based on the second expansion factor to determine the area of ​​interest corresponding to the first total feature map; and control the classification head to perform a third convolution operation on the first total feature map based on the second expansion factor to determine the classification detection result of the first total feature map.

[0099] During implementation, for the regression head: based on the second expansion factor, the third convolution kernel is used to perform a convolution operation on the first total feature map to obtain a second sub-feature map under the receptive field corresponding to the second expansion factor; and the fourth convolution kernel is used to perform a convolution operation on the second sub-feature map to obtain a second total feature map corresponding to the second sub-feature map; and the fifth convolution kernel is used to perform a convolution operation on the second total feature map to obtain a region of interest.

[0100] Specifically, the above-mentioned first total feature map (N, 128, 25, 25) is taken as an input item, and based on a second expansion factor with a value of 1, a convolution operation is performed using a third convolution kernel with a dimension of 128 and a size of 3×3 to obtain N×128 second sub-feature maps, that is, the second sub-feature map of (N, 128, 25, 25). The second sub-feature map (N, 128, 25, 25) is convolved with 128 kinds of 128-dimensional, 1×1-sized fourth convolution kernels to obtain the second total feature map under the receptive field corresponding to the second expansion factor, and then the dimension is upgraded to 512 dimensions. The second total feature map (N, 512, 25, 25) is convolved with the fifth convolution kernel (N, (4+1)*9, 512, 3, 3) of 3x3 size to obtain the second total feature map (N, (4+1)*9, 3, 3), and (N, 1*9, 25, 25) is passed to the classification head residual connection, and the remaining (N, 4*9, 25, 25) outputs the region of interest. Among them, "4" represents the position of the region of interest, "9" represents the number of regions of interest, and "1" represents the existence of an object in the feature map.

[0101] For the classification head: based on the second expansion factor, the sixth convolution kernel is used to perform a convolution operation on the first total feature map to obtain a third sub-feature map under the receptive field corresponding to the second expansion factor; and the seventh convolution kernel is used to perform a convolution operation on the third sub-feature map to obtain a third total feature map corresponding to the third sub-feature map; and the eighth convolution kernel is used to perform a convolution operation on the third total feature map to obtain a classification detection result.

[0102] Specifically, the above-mentioned first total feature map (N, 128, 25, 25) is taken as an input item, and based on the second expansion factor with a value of 1, a convolution operation is performed using a sixth convolution kernel with a dimension of 128 and a size of 3×3 to obtain N×128 third sub-feature maps, that is, the third sub-feature map of (N, 128, 25, 25). The third sub-feature map of (N, 128, 25, 25) is convolved with 128 kinds of 128-dimensional, 1×1-sized seventh convolution kernels to obtain the third total feature map under the receptive field corresponding to the second expansion factor, and then it is upgraded to 512 dimensions, and the third total feature map (N, 512, 25, 25) is convolved with the eighth convolution kernel (N, 9*34, 512, 3, 3) of 3x3 size to obtain the third total feature map of (N, 9*34, 3, 3); then the (N, 1*9, 25, 25) obtained by the regression head is expanded to (N, 1*9*34, 25, 25), and the residual is connected to the above-obtained (N, 9*34, 3, 3), and the most likely 9 categories (class) are obtained after the convolution operation. It should be noted here that the experimental data of this application is trained based on mahjong tiles, and there are 9 recognition possibilities, corresponding to the above-mentioned "9". That is, 9 types of image content that can be recognized by neural networks. Mahjong tiles have 34 classification results, corresponding to the above "34", which are the classifications that can be recognized by the neural network (such as "ten thousand", "two cakes", "Dongfeng", etc.). This process is equivalent to first performing a preliminary classification test on the image content of the third total feature map, first screening out the third total feature map that can be recognized by the neural network model (that is, the above 9 possibilities), and then classifying and identifying the region of interest of the screened third total feature map according to the position of the region of interest obtained by the regression head, so as to determine the preset category to which the third total feature map belongs (that is, which category it specifically belongs to among the 34 mahjong tile classifications). The preset classification is the final recognition result of the image to be recognized.

[0103] It should also be noted that the third, fourth, and fifth convolution kernels are used to screen out the second total feature map that can be recognized by the neural network, and the sixth, seventh, and eighth convolution kernels are used to determine the location of the region of interest. Although the dimensions and sizes of the third and fourth convolution kernels are the same as those of the first and second convolution kernels, and the dimensions and sizes of the sixth and seventh convolution kernels are also the same as those of the first and second convolution kernels, the internal configuration parameters used to determine the specific functions of the convolution kernels are not the same, and their internal configuration parameters can be set based on actual needs, and this application does not limit this.

[0104] In the decoder proposed by Yolof, for the classification head, the classification head first performs a complete dilated convolution operation on the (N, 512, 25, 25) feature map obtained by the encoder using a dilated convolution with a dilation factor of 1, and then repeats the above complete dilated convolution operation steps, for a total of 2 complete dilated convolution operations with a dilation factor of 1. The feature map after the operation is then classified using a standard 3x3 convolution, and the result is then residually connected with the (N, (4+1)*9, 25, 25) feature map obtained by the regression head to obtain the category on the (N, 9, 25, 25) feature map. For the regression head (Regression), the (N, 512, 25, 25) feature map obtained by the above Encoder is first subjected to a complete dilated convolution operation using a dilated convolution with a dilation factor of 1, and then the above complete dilated convolution operation steps are repeated 3 times, for a total of 4 complete dilated convolution operations with a dilation factor of 1, and then a standard 3x3 convolution is performed for feature regression to obtain a (N, (4+1)*9, 25, 25) feature map, of which (N, 1*9, 25, 25) is passed to the classification head residual connection, and the remaining (N, 4*9, 25, 25) outputs the region of interest.

[0105] It should be noted here that based on any expansion factor, the use of different convolution kernels to perform convolution operations represents in the neural network structure that a complete expansion convolution operation is performed through a dilated convolution block. Taking the regression head of the present application as an example, based on the second expansion factor, the entire process of using the third, fourth, and fifth convolution kernels to perform convolution operations is equivalent to the operation performed by a dilated convolution block. It can be seen that the classification head of Yolof mentioned above needs to complete two expansion convolution operations with an expansion factor of 1, and its regression head needs to complete four expansion convolution operations with an expansion factor of 1, that is, it represents that there are two expansion convolution blocks in the classification head of Yolof. In the above process of the present application, both the classification head and the regression head perform a dilated convolution operation based on the second expansion factor, that is, only one dilated convolution block needs to be set in the classification head and the regression head of the present application.

[0106] The network structure of this application is as follows Figure 3b As shown in the figure, first the image to be recognized is input into RegNetX for feature extraction, and then the extracted feature map to be processed is modified based on the encoder process in Yolof, and the convolution operation that originally used a 3×3 convolution kernel to perform 4 dilated convolutions is modified to a convolution operation based on E 1 To E 4(i.e., the first expansion factor of 4 values), respectively use the first convolution kernel of 3×3 size and the second convolution kernel of 1×1 size to perform the first convolution operation 4 times, so as to obtain the first total feature map. Finally, the first total feature map is input into the regression head and classification head of the decoder respectively. Among them, the regression head is used to determine the region of interest to be identified, and the classification head is used to classify and identify the image content in the region of interest to obtain the category to which the image to be identified belongs.

[0107] Next, we will use mahjong tiles as a training set to verify the feasibility of the technical solution of this application. First, we obtain a mahjong tile dataset, annotate and clean the mahjong tile dataset, and divide it into a training set and a test set. We obtain the feature map to be processed of the target dataset through feature extraction, and perform feature extraction and fusion on the feature map to be processed through strided stacked dilated convolution to enhance the small target detection performance. Specifically, the above dataset and its corresponding sample labels are randomly shuffled, and the training set and test set are divided according to the principle of 8:2.

[0108] The above training set and the corresponding sample labels are used as input data to train the neural network, and performance evaluation is performed in each training round. After the performance evaluation index reaches convergence, a neural network model with the ability to detect mahjong tiles is obtained. Furthermore, the test data set is tested to obtain the test results. Specifically, RegNetX-400MF is selected as the basis for constructing a feature extraction network, which contains 4 modules. The feature map output by each module is 2 times smaller than the feature map output by the previous module, and the downsampling is completed by increasing the convolution step size. Each module is composed of a different number of convolution blocks cascaded, and the numbers are [1,2,7,12] respectively, forming a residual network structure. Each convolution layer is followed by a Batch Normalization layer and a ReLU activation layer. The Batch Normalization layer can normalize the output, and the ReLU layer adds nonlinearity to the output of the upper layer. The original image size is obtained through feature extraction The feature map of the input size is extracted and fused through strided stacked dilated convolution to enhance the performance of dense target detection, and finally a feature map of the same size as the input feature map is obtained. Then, two sets of convolution modules are used to perform target classification and position prediction respectively, and map them to the original image, and output accurate mahjong tile target detection results through non-maximum suppression. Finally, the training set and the corresponding sample labels are used as input data to train the neural network. The input network size is not fixed, and the maximum size of each batch of images is read as the input size of the batch. Images smaller than the maximum size are expanded by adding patches. During the training process, training is performed until the performance evaluation indicators reach convergence to obtain the mahjong tile target detection model. The above neural network model is used to extract features from the test set. The prediction is made on the feature map, and the accurate target category and position coordinates are finally obtained through non-maximum suppression. Finally, the Mahjong tile target detector is trained using the target detection toolbox (MMDetection) deep learning framework.

[0109] According to experimental measurements, only one dilated convolution block is required in the classification head and regression head of the decoder of the present application to obtain a recognition accuracy close to that of Yolof.

[0110] Based on the same inventive concept, the present application embodiment also provides an image recognition device 400 based on deep learning, specifically as follows Figure 4 As shown, including:

[0111] The feature extraction module 401 is configured to perform feature extraction on the image to be identified, and obtain a plurality of feature maps to be processed representing different dimensions of the image to be identified; wherein the sizes of the feature maps to be processed are the same;

[0112] The feature fusion module 402 is configured to perform multiple rounds of first convolution operations on each feature map to be processed based on a preset first dilation factor, so as to determine a first total feature map corresponding to each feature map to be processed according to a feature recognition result obtained by each round of the first convolution operation; wherein the first dilation factor corresponding to each round of the first convolution operation is different, and different values ​​of the first dilation factor represent different receptive fields corresponding to the feature recognition result;

[0113] The image recognition module 403 is configured to determine the preset category to which the first total feature map belongs through a decoder, and use the preset category as the recognition result of the image to be recognized; wherein the first convolution operation process is as follows:

[0114] Based on the first expansion factor corresponding to this round, a first convolution kernel is used to perform a convolution operation on the input item to obtain a first sub-feature map corresponding to the input item; and a second convolution kernel is used to perform a convolution operation on the first sub-feature map to obtain a feature recognition result of the input item under the receptive field corresponding to the first expansion factor; wherein, the input item of the first round of the first convolution operation is the feature map to be processed, and the input item of the non-first round of the first convolution operation is the feature recognition result obtained in the previous round; the sizes of the first convolution kernel and the second convolution kernel are different.

[0115] In some possible embodiments, the preset dimensions of the first convolution kernel and the second convolution kernel are the same, and after performing the step of acquiring a plurality of feature maps to be processed representing different dimensions of the image to be identified, the feature extraction module 401 is further configured as follows:

[0116] Before performing multiple rounds of first convolution operations on each feature map to be processed, dimension processing is performed on each feature map to be processed so that the dimension of each feature map to be processed is the same as the preset dimension.

[0117] In some possible embodiments, the number of the second convolution kernels is greater than the number of the first convolution kernels, and the feature fusion module 402 is configured to:

[0118] Performing a convolution operation on each image in the input item to obtain a first sub-feature map corresponding to each image; wherein the image is a feature map to be processed or a feature recognition result;

[0119] To perform the convolution operation on the first sub-feature map using the second convolution kernel, the feature fusion module 402 is configured as follows:

[0120] A convolution operation is performed on each of the images in the input item, and pixel values ​​at the same position of each of the images are added during the convolution process to obtain feature extraction results corresponding to all the images in the input item.

[0121] In some possible embodiments, the image recognition module 403 is configured to:

[0122] Controlling the regression head to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map; and,

[0123] Controlling the classification head to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map; wherein the second expansion factor has a different value from the first expansion factor;

[0124] If the classification detection result indicates that the first total feature map is a recognizable image, category recognition is performed on the region of interest of the first total feature map, and the preset category to which the first total feature map belongs is determined according to the recognition result.

[0125] In some possible embodiments, the control regression head is executed to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map, and the image recognition module 403 is configured as follows:

[0126] Based on the second expansion factor, using a third convolution kernel to perform a convolution operation on the first total feature map to obtain a second sub-feature map under a receptive field corresponding to the second expansion factor; wherein the third convolution kernel has the same size as the first convolution kernel;

[0127] A fourth convolution kernel is used to perform a convolution operation on the second sub-feature map to obtain a second total feature map corresponding to the second sub-feature map; and a fifth convolution kernel is used to perform a convolution operation on the second total feature map to obtain the region of interest; wherein the sizes of the third convolution kernel and the fifth convolution kernel are the same as those of the first convolution kernel; the fourth convolution kernel is the same as the size of the second convolution kernel, and the number of the fourth convolution kernels is greater than the number of the third convolution kernels.

[0128] In some possible embodiments, the control classification head is executed to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map, and the image recognition module 403 is configured as follows:

[0129] Based on the second expansion factor, using a sixth convolution kernel to perform a convolution operation on the first total feature map to obtain the third sub-feature map under the receptive field corresponding to the second expansion factor; wherein the sixth convolution kernel has the same size as the first convolution kernel;

[0130] A seventh convolution kernel is used to perform a convolution operation on the third sub-feature map to obtain a third total feature map corresponding to the third sub-feature map; and an eighth convolution kernel is used to perform a convolution operation on the third total feature map to obtain the classification detection result; wherein, the sizes of the sixth convolution kernel and the eighth convolution kernel are the same as those of the first convolution kernel; the seventh convolution kernel is the same as the size of the second convolution kernel, and the number of the seventh convolution kernels is greater than the number of the sixth convolution kernels.

[0131] In some possible embodiments, the classification head and the regression head in the decoder each include at least one dilated convolution block.

[0132] Refer to the following Figure 5The electronic device 130 according to this embodiment of the present application is described. Figure 5 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Figure 5 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0134] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0135] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0136] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0137] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions, and the instructions can be executed by the processor 131 of the device 400 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0139] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, which, when executed by the processor 131, implements an image recognition method based on deep learning as provided in the present application.

[0140] In an exemplary embodiment, various aspects of a deep learning-based image recognition method provided by the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a deep learning-based image recognition method according to various exemplary embodiments of the present application described above in this specification.

[0141] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0142] The program product for image recognition based on deep learning of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0143] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0145] The program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "like" or similar programming languages. The program code may be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device may be connected to the user electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0146] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0147] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0153] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A deep learning-based image recognition method, characterized in that: The method comprises: Perform feature extraction on the image to be identified, and obtain a plurality of feature maps to be processed representing different dimensions of the image to be identified; wherein the sizes of the feature maps to be processed are the same; Based on a preset first dilation factor, multiple rounds of first convolution operations are performed on each feature map to be processed, so as to determine a first total feature map corresponding to each feature map to be processed according to a feature recognition result obtained by each round of the first convolution operation; wherein the first dilation factor corresponding to each round of the first convolution operation is different, and different values ​​of the first dilation factor represent different receptive fields corresponding to the feature recognition result; The preset category to which the first total feature map belongs is determined by a decoder, and the preset category is used as the recognition result of the image to be recognized; wherein the first convolution operation process is as follows: Based on the first dilation factor corresponding to this round, a first convolution kernel is used to perform a convolution operation on the input item to obtain a first sub-feature map corresponding to the input item; and a second convolution kernel is used to perform a convolution operation on the first sub-feature map to obtain a feature recognition result of the input item under the receptive field corresponding to the first dilation factor; wherein the input item of the first convolution operation of the first round is the feature map to be processed, and the input item of the first convolution operation other than the first round is the feature recognition result obtained in the previous round; the sizes of the first convolution kernel and the second convolution kernel are different; The step of determining the preset category to which the first total feature map belongs by a decoder includes: Controlling the regression head to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map; and, Controlling the classification head to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map; wherein the second expansion factor has a different value from the first expansion factor; If the classification detection result indicates that the first total feature map is a recognizable image, category recognition is performed on the region of interest of the first total feature map, and the preset category to which the first total feature map belongs is determined according to the recognition result.

2. The method according to claim 1, characterized in that The preset dimensions of the first convolution kernel and the second convolution kernel are the same. After acquiring a plurality of feature maps to be processed that represent different dimensions of the image to be identified, the method further includes: Before performing multiple rounds of first convolution operations on each feature map to be processed, dimension processing is performed on each feature map to be processed so that the dimension of each feature map to be processed is the same as the preset dimension.

3. The method according to claim 2, characterized in that The number of the second convolution kernels is greater than the number of the first convolution kernels, and the using the first convolution kernel to perform a convolution operation on the input item includes: Performing a convolution operation on each image in the input item to obtain a first sub-feature map corresponding to each image; wherein the image is a feature map to be processed or a feature recognition result; The using a second convolution kernel to perform a convolution operation on the first sub-feature map includes: A convolution operation is performed on each of the images in the input item, and pixel values ​​at the same position of each of the images are added during the convolution process to obtain feature extraction results corresponding to all the images in the input item.

4. The method according to claim 1, characterized in that The controlling regression head performs a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map, including: Based on the second expansion factor, using a third convolution kernel to perform a convolution operation on the first total feature map to obtain a second sub-feature map under a receptive field corresponding to the second expansion factor; wherein the third convolution kernel has the same size as the first convolution kernel; A fourth convolution kernel is used to perform a convolution operation on the second sub-feature map to obtain a second total feature map corresponding to the second sub-feature map; and a fifth convolution kernel is used to perform a convolution operation on the second total feature map to obtain the region of interest; wherein the sizes of the third convolution kernel and the fifth convolution kernel are the same as those of the first convolution kernel; the fourth convolution kernel is the same as the size of the second convolution kernel, and the number of the fourth convolution kernels is greater than the number of the third convolution kernels.

5. The method according to claim 1, characterized in that The controlling classification head performs a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map, including: Based on the second expansion factor, using a sixth convolution kernel to perform a convolution operation on the first total feature map to obtain a third sub-feature map under a receptive field corresponding to the second expansion factor; wherein the sixth convolution kernel has the same size as the first convolution kernel; A seventh convolution kernel is used to perform a convolution operation on the third sub-feature map to obtain a third total feature map corresponding to the third sub-feature map; and an eighth convolution kernel is used to perform a convolution operation on the third total feature map to obtain the classification detection result; wherein, the sizes of the sixth convolution kernel and the eighth convolution kernel are the same as those of the first convolution kernel; the seventh convolution kernel is the same as the size of the second convolution kernel, and the number of the seventh convolution kernels is greater than the number of the sixth convolution kernels.

6. The method according to any one of claims 1 to 5, characterized in that: The classification head and the regression head in the decoder each include at least one dilated convolution block.

7. An image recognition device based on deep learning, characterized in that: The device comprises: A feature extraction module is configured to perform feature extraction on the image to be identified, and obtain a plurality of feature maps to be processed representing different dimensions of the image to be identified; wherein the sizes of the feature maps to be processed are the same; The feature fusion module is configured to perform multiple rounds of first convolution operations on each feature map to be processed based on a preset first dilation factor, so as to determine a first total feature map corresponding to each feature map to be processed according to a feature recognition result obtained by each round of the first convolution operation; wherein the first dilation factor corresponding to each round of the first convolution operation is different, and different values ​​of the first dilation factor represent different receptive fields corresponding to the feature recognition result; The image recognition module is configured to determine the preset category to which the first total feature map belongs through a decoder, and use the preset category as the recognition result of the image to be recognized; wherein the first convolution operation process is as follows: Based on the first dilation factor corresponding to this round, a first convolution kernel is used to perform a convolution operation on the input item to obtain a first sub-feature map corresponding to the input item; and a second convolution kernel is used to perform a convolution operation on the first sub-feature map to obtain a feature recognition result of the input item under the receptive field corresponding to the first dilation factor; wherein the input item of the first convolution operation of the first round is the feature map to be processed, and the input item of the first convolution operation other than the first round is the feature recognition result obtained in the previous round; the sizes of the first convolution kernel and the second convolution kernel are different; Wherein, the image recognition module is configured to: Controlling the regression head to perform a second convolution operation on the first total feature map based on a second dilation factor to determine a region of interest corresponding to the first total feature map; and, Controlling the classification head to perform a third convolution operation on the first total feature map based on the second expansion factor to determine a classification detection result of the first total feature map; wherein the second expansion factor has a different value from the first expansion factor; If the classification detection result indicates that the first total feature map is a recognizable image, category recognition is performed on the region of interest of the first total feature map, and the preset category to which the first total feature map belongs is determined according to the recognition result.

8. An electronic device, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-6.

9. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 6.

Images