Digital recognition method, device, terminal device and storage medium

Abstract

This application is applicable to the field of image recognition technology, and provides a digital recognition method, device, terminal device, and storage medium. The method includes: inputting a to-be-recognized image containing digits into multiple convolutional layers of a digital recognition network model; according to the processing order of the multiple convolutional layers, sequentially passing the to-be-recognized image through each convolutional layer for forward convolution processing and inverse convolution processing to obtain multiple inverse image features; respectively inputting the multiple inverse image features into a classifier of the digital recognition network model to recognize the digits contained in the to-be-recognized image. By using the above method to perform forward convolution processing and inverse convolution processing on the to-be-recognized image in sequence, multiple inverse image features in different dimensions are obtained, enabling it to contain more feature information of the to-be-recognized image. Furthermore, the accuracy of the digital recognition network model in recognizing the to-be-recognized image based on the multiple inverse image features is improved.

Description

Technical Field

[0001] This application belongs to the technical field of image recognition, and particularly relates to a digital recognition method, device, terminal device, and storage medium. Background Art

[0002] In the prior art, one of the main tasks of substation inspection robots is to replace human operators for on-site inspections and intelligently identify whether the operating parameters of equipment are normal. The commonly used method is: the inspection robot takes images of the equipment, and then a neural network model is used to perform digital recognition on the collected images. However, when the existing neural network model extracts image features, it usually performs convolution processing on the image to obtain the features output by the last convolution layer, and digital recognition is performed based on these features. However, the features obtained by this method cannot contain more detailed information in the image, resulting in a low digital recognition accuracy when performing digital recognition on the image. Summary of the Invention

[0003] Embodiments of this application provide a digital recognition method, device, terminal device, and storage medium, which can solve the problem of low digital recognition accuracy in the prior art when performing digital recognition on images.

[0004] In a first aspect, embodiments of this application provide a digital recognition method, including:

[0005] Input an image to be recognized containing numbers into a digital recognition network model, where the digital recognition network model includes multiple convolutional layers;

[0006] According to the forward processing order of the multiple convolutional layers, sequentially perform forward convolution processing on the image to be recognized through each convolutional layer to obtain multiple forward image features corresponding to each convolutional layer one by one, where the input image of the first convolutional layer is the image to be recognized, and the input image of each remaining convolutional layer is the forward image feature output by the adjacent upper convolutional layer;

[0007] According to the reverse processing order of the multiple convolutional layers, sequentially perform reverse convolution processing on the forward image feature of the last layer through each convolutional layer to obtain multiple reverse image features corresponding to each convolutional layer one by one, where the reverse image feature of the last layer is the forward image feature of the last layer, and the input image of each remaining convolutional layer is the reverse image feature output by the adjacent upper convolutional layer;

[0008] Input the reverse image features of the multiple convolutional layers into the classifier of the digital recognition network model respectively to recognize the numbers contained in the image to be recognized.

[0009] In one embodiment, according to the reverse processing order of the multi-layer convolutional layers, the forward image features of the last layer are sequentially subjected to reverse convolution processing through each convolutional layer to obtain a plurality of reverse image features corresponding to each convolutional layer, including:

[0010] For any convolutional layer, obtain the forward image features of the current convolutional layer and the reverse image features output by the previous convolutional layer adjacent to the current convolutional layer;

[0011] Perform reverse convolution processing on the forward image features and the reverse image features output by the previous convolutional layer to obtain the reverse image features of the current convolutional layer.

[0012] In one embodiment, the plurality of reverse image features include the first reverse image feature of the first convolutional layer; according to the reverse processing order of the multi-layer convolutional layers, the forward image features of the last layer are sequentially subjected to reverse convolution processing through each convolutional layer to obtain a plurality of reverse image features corresponding to each convolutional layer, including:

[0013] Obtain the second reverse image feature of the second convolutional layer and the third reverse image feature of the third convolutional layer in the multi-layer convolutional layers;

[0014] Perform feature fusion on the second reverse image feature and the third reverse image feature to obtain a target feature;

[0015] Perform reverse convolution processing on the target feature and the forward image features of the first convolutional layer to obtain the first reverse image feature of the first convolutional layer.

[0016] In one embodiment, the digital recognition network model includes a semantic convolution module; the performing feature fusion on the second reverse image feature and the third reverse image feature to obtain a target feature includes:

[0017] Input the third reverse image feature into the semantic convolution module for feature processing to obtain the semantic information features in the third reverse image feature;

[0018] Perform feature fusion on the second reverse image feature and the semantic information features to obtain an initial target feature;

[0019] Perform feature selection on the initial target feature according to the semantic information features to obtain the target feature.

[0020] In one embodiment, the inputting the third reverse image feature into the semantic convolution module for feature processing to obtain the semantic information features in the third reverse image feature includes:

[0021] The third reverse image feature is successively subjected to feature processing through the semantic convolution layer and the activation layer in the semantic convolution module to obtain an initial semantic information feature;

[0022] The initial semantic information feature and the third reverse image feature are subjected to feature fusion to obtain a fusion feature;

[0023] The fusion feature is subjected to reverse convolution processing to obtain the semantic information feature.

[0024] In one embodiment, before performing feature selection on the initial target feature according to the semantic information feature to obtain the target feature, it includes:

[0025] A preset activation function is used to perform strengthening processing on the semantic information feature to generate the strengthened semantic information feature.

[0026] In one embodiment, the digital recognition network model is obtained by training the model using target training images; the target training images are obtained through the following steps:

[0027] Obtain a digital template image in a target format;

[0028] Perform a target processing operation on the digital template image to obtain a target template image, and the target processing operation includes at least one of cropping, rotating, translating, and denoising;

[0029] Delete images with a similarity lower than a preset threshold from the target template image to obtain the target training image.

[0030] In a second aspect, an embodiment of the present application provides a digital recognition device, including:

[0031] An input module, configured to input a to-be-recognized image containing numbers into a digital recognition network model, and the digital recognition network model includes multiple convolutional layers;

[0032] A first processing module, configured to sequentially perform forward convolution processing on the to-be-recognized image through each convolutional layer according to the forward processing order of the multiple convolutional layers to obtain a plurality of forward image features corresponding to each convolutional layer one by one, where the input image of the first convolutional layer is the to-be-recognized image, and the input image of each of the remaining convolutional layers is the forward image feature output by the adjacent upper convolutional layer;

[0033] A second processing module, configured to, according to the reverse processing order of the multi-layer convolutional layers, sequentially perform reverse convolutional processing on the forward image features of the last layer through each convolutional layer to obtain a plurality of reverse image features corresponding to each convolutional layer one by one, where the reverse image feature of the last layer is the forward image feature of the last layer, and the input image of each of the remaining convolutional layers is the reverse image feature output by the adjacent upper convolutional layer;

[0034] An identification module, configured to input the reverse image features of the multi-layer convolutional layers into the classifier of the digital recognition network model respectively to identify the digits included in the image to be recognized.

[0035] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where when the processor executes the computer program, the method described in any one of the above first aspects is implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, where when the computer program is executed by a processor, the method described in any one of the above first aspects is implemented.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product, which when running on a terminal device causes the terminal device to execute the method described in any one of the above first aspects.

[0038] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: By first performing multi-layer forward convolutional processing on the input image to be recognized, the digital recognition network model can gradually extract more and more complex high-dimensional forward image features from the low-dimensional features of the image to be recognized. Then, the digital recognition network model can perform reverse convolutional processing on the forward image features to obtain a plurality of reverse image features to restore the high-dimensional forward image features. So that the obtained plurality of reverse image features can contain more detailed feature information in the image to be recognized, which is beneficial for the digital recognition network model to accurately obtain the classification information of each pixel point in the plurality of restored reverse image features when performing pixel prediction processing on the plurality of reverse image features. Furthermore, the accuracy of the digital recognition network model inside the terminal device in recognizing the digits included in the image to be recognized can be improved. Description of the Drawings

[0039] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0040] Figure 1 It is a flowchart of the implementation of a digital recognition method provided by an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of the network framework structure of a digital recognition network model applicable to a digital recognition method provided by an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of an implementation manner of S103 of a digital recognition method provided by an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of an implementation manner of S103 of a digital recognition method provided by another embodiment of the present application;

[0044] Figure 5 It is a schematic diagram of an implementation manner of S1034 of a digital recognition method provided by an embodiment of the present application;

[0045] Figure 6 It is a schematic diagram of the network framework structure for feature fusion processing of the second reverse image feature and the third reverse image feature in a digital recognition network model applicable to a digital recognition method provided by an embodiment of the present application;

[0046] Figure 7 It is a flowchart of the implementation of a digital recognition method provided by another embodiment of the present application;

[0047] Figure 8 It is a block diagram of the structure of a digital recognition device provided by an embodiment of the present application; <l

[0048] Figure 9 It is a block diagram of the structure of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0049] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0050] It should be understood that when used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0051] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0052] The digital recognition method provided by the embodiments of this application can be applied to terminal devices such as mobile phones, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, etc. In addition, the above digital recognition method can also be applied to inspection robots. For example, the digital recognition method can be applied to inspection robots for power distribution rooms, inspection robots for substations, etc., to accurately recognize the numbers on the power digital meters in electrical equipment. The specific types of terminal devices and inspection robots are not limited in the embodiments of this application.

[0053] Please refer to Figure 1 , Figure 1 , which shows an implementation flowchart of a digital recognition method provided by the embodiments of this application. The method includes the following steps:

[0054] S101. Input an image to be recognized containing numbers into a digital recognition network model, where the digital recognition network model includes multiple convolutional layers.

[0055] In applications, the above image to be recognized can be an image obtained by a terminal device in real time during an inspection of a power digital meter, or an image pre-stored inside the terminal device, which is not limited herein. The above digital recognition network model can be a model pre-trained by a user using training data, which is used to perform feature processing on the input image to be recognized and then output a prediction result. Among them, the digital recognition network model includes multiple convolutional layers, and each convolutional layer can be composed of several identical or different convolutional units to perform convolutional operations on the input image to be recognized. Among them, the purpose of the convolutional operation is to extract different features of the input image to be recognized. It should be added that using only one convolutional layer to perform convolutional processing on the image to be recognized may only extract some low-level features. For example, features at the levels of edges, lines, and corners of the image to be recognized. Therefore, multiple convolutional layers can be set to process the image to be recognized, so that the digital recognition network model can gradually extract more and more complex features from low-dimensional features.

[0056] It should be noted that the above digital recognition network model can be composed of multiple neural network models, and each neural network model can include at least one convolutional layer for performing convolutional processing on the feature image, and there is no limitation in this regard.

[0057] S102. According to the forward processing order of the multi-layer convolutional layers, sequentially pass the image to be recognized through each convolutional layer for forward convolutional processing to obtain multiple forward image features corresponding to each convolutional layer one by one, where the input image of the first convolutional layer is the image to be recognized, and the input image of each remaining convolutional layer is the forward image feature output by the adjacent upper convolutional layer.

[0058] In application, the forward processing order of the above multi-layer convolutional layers is the order in which the digital recognition network model processes the input image to be recognized. It should be added that the dimension and eigenvalue of the forward image feature output by each convolutional layer are both determined by the convolutional kernel and convolutional step of the current convolutional layer. However, the convolutional kernel and convolutional step of each above-mentioned convolutional layer can be set by the user when training the digital recognition network model.

[0059] In application, the above forward convolutional processing can be considered as a process of performing convolutional operations on the input image through the convolutional kernel and convolutional step of each convolutional layer. Among them, the convolutional operation can be considered as a process of weighted summation, where each pixel in the region of the input image to be recognized is multiplied by each element of the convolutional kernel (weight matrix) respectively, and the sum of all products is used as the new value of the pixel, thereby obtaining the forward image feature. The convolutional kernel is the weight used in convolution and can be represented by a matrix (weight matrix). Among them., a feature matrix can be established according to each pixel in the region of the image to be recognized to participate in the above convolutional operation.

[0060] Exemplarily, referring to Figure 2 , it can be considered that Figure 2 the 512*512-sized image in is the image to be recognized input by the terminal device. Then, the digital recognition network model sequentially processes the image to be recognized through multiple convolutional layers to obtain multiple forward image features. For example, the multiple forward image features obtained are Figure 2 the images of 256*256, 128*128, 64*64, 32*32, 16*16, etc. in. From the above Figure 2 it can be seen that each forward image feature is obtained by the current convolutional layer performing convolutional processing on the forward image feature output by the adjacent upper convolutional layer. At this time, the forward processing order of the above multi-layer convolutional layers is Figure 2 the order of obtaining the above multiple forward image features in (the arrow order from the image to be recognized 512*512 to the forward image feature 16*16).

[0061] It can be understood that the numbers 512, 256, 128, 64, 32, and 16 mentioned above are all the sizes of the input images for each layer. Among them, the input image for each layer can be understood as an image feature matrix of the corresponding size. Exemplarily, for an input image with a size of 16*16, an image feature matrix of 16*16 can be constructed, and each matrix value in the matrix can correspond to the pixel value of each pixel point in the input image. In addition, the dimensions of the output forward image features (the width W and length H of the input image for each layer) can be calculated by the following formula. Specifically, W1 or H1 = [(input image size (W2 or H2) - convolution kernel size) / convolution stride] + 1. Among them, when calculating W1 of the forward image features, the input image size is the size of W2 of the input image of the convolutional layer; and when calculating H1 of the forward image features, the input image size is the size of H2 of the input image of the convolutional layer.

[0062] S103. According to the reverse processing order of the multi-layer convolutional layers, successively perform reverse convolution processing on the forward image features of the last layer through each convolutional layer to obtain a plurality of reverse image features corresponding to each convolutional layer one by one, where the reverse image feature of the last layer is the forward image feature of the last layer, and the input image of each of the remaining convolutional layers is the reverse image feature output by the adjacent upper convolutional layer.

[0063] In applications, the forward processing order of the multi-layer convolutional layers has been explained in S102 above, and it can be considered that the above reverse processing order is the opposite of the forward processing order. That is, the reverse processing order starts from the last convolutional layer to process the input image first, and the first convolutional layer processes the input image last. It should be particularly noted that the last convolutional layer in the reverse processing order is the same convolutional layer as the last layer in the forward processing order. In addition, the reverse image feature of the last convolutional layer is the forward image feature output by the last convolutional layer, and the input image of each of the remaining convolutional layers is the reverse image feature output by the adjacent upper convolutional layer. At this time, when performing reverse convolution processing, the upper convolutional layer of the current convolutional layer is the lower convolutional layer of the current convolutional layer during forward convolution processing. Exemplarily, referring to Figure 2 , if the current convolutional layer is the convolutional layer that outputs reverse image features of 32*32, then when performing reverse convolution processing, its upper convolutional layer is the convolutional layer that outputs reverse image features of 16*16. And when performing forward convolution processing, if the current convolutional layer is the convolutional layer that outputs forward image features of 32*32, then when performing forward convolution processing, its lower convolutional layer is the convolutional layer that outputs forward image features of 16*16.

[0064] In applications, referring to Figure 2For the dashed box part, the forward image feature output by the last convolutional layer in the forward convolutional process is a 16*16 image feature. At this time, the 16*16 forward image feature is used as the input image and input into the last inverse convolutional layer, and the 16*16 inverse image feature (the inverse image feature of the last layer) in the dashed box has an upward output arrow. That is to say, the 16*16 forward image feature is both the input image of the last convolutional layer and the output for the inverse image feature of the last convolutional layer. Based on this, when obtaining the penultimate layer (the 32*32 inverse image feature), the input image of this convolutional layer includes the 16*16 inverse image feature.

[0065] It should be added that the above inverse convolution also needs to use the corresponding convolution kernel and convolution step to perform inverse convolution on the input feature image. In addition, the convolution kernel and convolution step of each convolutional layer can also be set by the user when training the digital recognition network model, and they can also be the same as the convolution kernel and convolution step of each convolutional layer during the forward convolution process, and this is not limited.

[0066] Exemplarily, if the input image is a feature image with a size of 16*16, first, upsampling can be used to make the input image become 32*32. At this time, each pixel point in the 16*16 of the original input image needs to be retained in the upsampled input image, and then multiple other blank pixel points are added between the original pixel points (the pixel values of the added pixel points can be set by the user in advance according to the actual situation). Then, the upsampled input image is convolved using a preset convolution kernel and convolution step to obtain the required inverse image feature (the 32*32 inverse image feature).

[0067] It should be noted that in traditional image recognition network models, usually only forward convolution processing is performed on the input image. Correspondingly, the size of the output feature image will decrease; however, the significance of inverse convolution processing is to restore the small-sized feature image so that the neural network model can perform pixel prediction processing on the restored feature image to obtain the classification information of each pixel point in the feature image. At this time, the restored feature image is not the same as the input image, and the restored feature image can be understood as the important information retained after processing the input image. In this way, the terminal device can achieve the purpose of accurately recognizing the numbers contained in the image to be recognized.

[0068] S104. Input the inverse image features of the multiple convolutional layers into the classifier of the digital recognition network model respectively to recognize the numbers contained in the image to be recognized.

[0069] In an application, after obtaining the above reverse image features, each reverse image feature can be input into a classifier simultaneously, so that the classifier can make predictions based on multiple reverse image features. Among them, the above classifier can be regarded as a classification module in a digital recognition network model. When training this classification module, the training data can be processed through the above steps S102 and 103 in sequence to obtain the reverse image features of the training data. Then, the processed reverse image features are input into the classification module to obtain the prediction result of the classification module. Finally, the classification error between the prediction result and the actual result is calculated, and the model parameters in the classification module are iterated according to the classification error until the final classification module is obtained.

[0070] In this embodiment, by first performing multi-layer forward convolution processing on the input image to be recognized, the digital recognition network model can gradually extract more and more complex high-dimensional forward image features from the low-dimensional features of the image to be recognized. Then, the digital recognition network model can perform reverse convolution processing on the forward image features to obtain multiple reverse image features, so as to restore the high-dimensional forward image features. The obtained multiple reverse image features can contain more detailed feature information in the image to be recognized, which is beneficial for the digital recognition network model to accurately obtain the classification information of each pixel point in the multiple restored reverse image features when performing pixel prediction processing on the multiple reverse image features. Furthermore, the accuracy of the digital recognition network model inside the terminal device in recognizing the numbers contained in the image to be recognized can be improved.

[0071] Please refer to Figure 3 , in a specific embodiment, S103 sequentially performs reverse convolution processing on the forward image features of the last layer through each convolutional layer according to the reverse processing order of the multi-layer convolutional layers, to obtain multiple reverse image features corresponding to each convolutional layer one by one. The specific steps include the following sub-steps S1031 - S1032, which are described in detail as follows:

[0072] S1031: For any convolutional layer, obtain the forward image features of the current convolutional layer and the reverse image features output by the previous convolutional layer adjacent to the current convolutional layer.

[0073] S1032: Perform reverse convolution processing on the forward image features and the reverse image features output by the previous convolutional layer to obtain the reverse image features of the current convolutional layer.

[0074] In an application, according to the explanation in S102 above, it can be known that the digital recognition network model can obtain the forward image features of each convolutional layer. In addition, it has been explained in S103 above that when performing inverse convolution processing, the previous convolutional layer of the current convolutional layer is the next convolutional layer during forward convolution processing. Based on this, the digital recognition network model can perform convolution processing according to both the forward image features of the current convolutional layer and the inverse image features output by the previous convolutional layer to obtain the inverse image features of the current convolutional layer.

[0075] Exemplarily, referring to Figure 2 , for the inverse image feature output by the previous convolutional layer being a feature image of 16*16 and the forward image feature being a feature image of 32*32, the inverse convolution processing can be first performed on the 16*16 feature image to make the size of the feature image after the inverse convolution processing the same as the size of the input image of 32*32. Then, the feature image after the inverse convolution processing is fused with the input image of 32*32, that is, the pixel values (or feature values) under the same pixel point are superimposed and fused to obtain a new input image of 32*32 in size. The inverse image feature of the current convolutional layer can be obtained by establishing a feature matrix based on the new input image of 32*32 in size.

[0076] It should be noted that when the digital recognition network model performs forward convolution on the input image, the size of the input image is gradually reduced, that is, some information in the input image is lost during the processing. Based on this, if only the forward image features of the last convolutional layer are subjected to inverse convolution processing to obtain the inverse image features of the current convolutional layer, the detailed information of the image to be recognized at different sizes cannot be obtained during the inverse convolution process. Based on this, the forward image features of the current layer and the inverse image features of the previous layer can be simultaneously subjected to inverse convolution processing, so that the obtained inverse image features of the current convolutional layer can contain more detailed information of the input image to be recognized.

[0077] Please refer to Figure 4 , in a specific embodiment, the multiple inverse image features include the first inverse image feature of the first convolutional layer; S103 sequentially performs inverse convolution processing on the forward image features of the last layer through each convolutional layer according to the inverse processing order of the multiple convolutional layers to obtain the multiple inverse image features corresponding to each convolutional layer one by one, specifically including the following sub-steps S1033 - S1035, which are detailed as follows:

[0078] S1033. Obtain the second inverse image feature of the second convolutional layer and the third inverse image feature of the third convolutional layer in the multiple convolutional layers.

[0079] In an application, each of the above convolutional layers has a corresponding forward processing order. Therefore, after the to-be-recognized image is processed through the steps of S102 and S103, the second reverse image feature of the second convolutional layer and the third reverse image feature of the third convolutional layer can be obtained.

[0080] In other applications, the terminal device can also obtain the reverse image features of the remaining convolutional layers. However, it should be understood that for the way of processing the to-be-recognized image by multiple convolutional layers, the layer closer to the input (the first convolutional layer) learns more general features in the to-be-recognized image, while the layer closer to the output (the last convolutional layer) learns more abstract features in the to-be-recognized image. Based on this, it can be considered that when obtaining the second reverse image feature of the second convolutional layer and the third reverse image feature of the third convolutional layer respectively, the third reverse image feature not only has the information of the reverse image features of each convolutional layer after the third convolutional layer, but also has the general features learned by the layer closer to the input (the first convolutional layer) (because the way of obtaining the reverse image feature includes the forward image feature of the third convolutional layer). Therefore, it can be considered that both the second reverse image feature and the third reverse image feature contain more detailed information of the to-be-input image.

[0081] S1034. Feature-fuse the second reverse image feature and the third reverse image feature to obtain a target feature.

[0082] In an application, as described above, the second reverse image feature and the third reverse image feature contain more detailed information of the to-be-input image. Based on this, the terminal device can feature-fuse the above two image features to obtain a target feature, so that the target feature can contain the feature information of the two image features. Among them, the feature-fusion of the second reverse image feature can be to perform reverse convolution on the third reverse image feature to make the dimension (the size of the feature image) of the third reverse image feature consistent with the dimension (the size of the feature image) of the second reverse image feature. Then, the pixel values (or feature values) under the same pixel point in the feature image are superimposed and fused to obtain a target fusion feature.

[0083] S1035. Perform reverse convolution processing on the target feature and the forward image feature of the first convolutional layer to obtain the first reverse image feature of the first convolutional layer.

[0084] In the application, it is recorded in the above S1032 that the forward image features and the reverse image features output by the previous convolutional layer are subjected to reverse convolution processing to obtain the reverse image features of the current convolutional layer. If the current convolutional layer is the first convolutional layer, the above S1032 is to perform reverse convolution processing on the forward image features of the first convolutional layer and the reverse image features output by the second convolutional layer to obtain the first reverse image features of the first convolutional layer. However, as described in the above S1033 and S1034, in order to enable the reverse image features output by the second convolutional layer to contain more feature information of the image to be recognized, the second reverse image features and the third reverse image features can be fused and processed as the input image of the first convolutional layer.

[0085] Specifically, reference can be made to Figure 2 , Figure 2 in which P1 is represented as the third reverse image feature, P2 is represented as the second reverse image feature, and P3 is the image feature obtained after processing according to the third reverse image feature and the second reverse image feature. Then, the reverse convolution processing is performed on the image feature (P3) and the forward image features of the first convolutional layer to obtain the first reverse image features. For example, only the reverse convolution processing is performed on the image feature (P3) to make its feature dimension consistent with the dimension (256*256) of the forward image features of the first convolutional layer, and then the feature fusion is performed. In this way, the obtained first reverse image features can further contain more feature information of the image to be recognized.

[0086] Please refer to Figure 5 , in a specific embodiment, the digital recognition network model includes a semantic convolution module; S1034 fuses the second reverse image features and the third reverse image features to obtain target features, which specifically includes the following sub-steps S10341-S10343, described in detail as follows:

[0087] S10341: Input the third reverse image features into the semantic convolution module for feature processing to obtain the semantic information features in the third reverse image features.

[0088] In the application, the above semantic convolution module may include a plurality of semantic convolutional layers pre-trained or set by the user, which are used to perform feature processing on the third reverse image features to obtain semantic information features. It should be noted that because the general features (the second reverse image features and the third reverse image features) learned by the above levels close to the input (the first convolutional layer). Based on this, it can be considered that the feature information learned by the third reverse image features is the semantic information of the image to be recognized, and the feature information learned by the second reverse image features is the regional texture information of the image to be recognized. Therefore, the third reverse image features can be input into the semantic convolution module for feature processing to further extract the semantic information features in the third reverse image features.

[0089] S10342. Feature fuse the second reverse image feature and the semantic information feature to obtain an initial target feature.

[0090] S10343. Feature select the initial target feature according to the semantic information feature to obtain the target feature.

[0091] In application, the above semantic information feature is obtained by feature processing the third reverse image feature. Therefore, the second reverse image feature and the semantic information feature can be feature fused to obtain an initial target feature. Among them, the way of feature fusing the second reverse image feature and the semantic information feature is similar to the way of feature fusing the second information feature and the third reverse feature. For specific explanation, please refer to the explanation in S1034 above, and no further description will be given here.

[0092] In application, the above feature selection refers to selecting N features from the existing M features (initial target features) to optimize the specific index of the system, which is a process of selecting some of the most effective features from the original features to reduce the dimension of the data set. It should be added that for the above initial target feature, it is obtained by feature fusing the second reverse image feature and the semantic information feature. Therefore, it can be considered that the above initial target feature is a feature matrix with a size of 128*128. Similarly, the above semantic information feature is obtained by feature processing the third reverse image feature through a semantic convolution module, and it can be considered as a feature image with a size of 64*64. Finally, the target feature matrix can be obtained by multiplying the two feature matrices, and the target feature image (i.e., the target feature) can be re-obtained according to the values in the target feature matrix.

[0093] In one embodiment, before S10344 feature selects the initial target feature according to the semantic information feature to obtain the target feature, the semantic information feature needs to be processed through the following steps, and the processed semantic information feature is used to feature select the initial target feature. Specifically:

[0094] Reinforce the semantic information feature by using a preset activation function to generate the reinforced semantic information feature.

[0095] In application, the above activation function can be an activation function preset by the user in the semantic convolution module. Or, it can also be considered that after obtaining the semantic information feature, the semantic information feature is input into the activation layer of the semantic convolution module, and the activation function of the activation layer is used to reinforce the semantic information feature.

[0096] In an application, the above activation functions include, but are not limited to, the Tanh activation function, the Sigmoid activation function, the ReLU activation function, etc., and are not limited thereto. In the application, after the above semantic information features are input into the activation function, the activation function can perform fixed mathematical operations on them, and the obtained feature values are the semantic information features. Specifically, after the above activation function processes the semantic information features, it can output the possibility (probability) of the current third reverse image feature in a certain category, and the probability value is generally between [0, 1]. Then, the semantic information features with probabilities lower than the preset threshold are deleted, and only the semantic information features with probabilities higher than the preset threshold are retained. In this way, enhanced semantic information features are generated. At this time, the enhanced semantic information features only retain the important features in the original semantic information features and ignore the unimportant features.

[0097] In a specific embodiment, S10341 inputs the third reverse image feature into the semantic convolution module for feature processing to obtain the semantic information features in the third reverse image feature, which specifically includes the following sub-steps and is described in detail as follows:

[0098] The third reverse image feature is sequentially subjected to feature processing through the semantic convolution layer and the activation layer in the semantic convolution module to obtain the initial semantic information features.

[0099] In the application, it has been described in the above S10341 that multiple semantic convolution layers are provided in the semantic convolution module for feature processing of the third reverse image feature. Specifically, referring to the above Figure 6 the content within the dotted box, the above semantic convolution module includes two semantic convolution layers (i.e., Figure 6 Conv3*3 in the figure), the size of the convolution kernel set in each convolution layer can be 3*3, the convolution step can be 1, and the specific values of each convolution kernel can also be set by the user according to the actual situation.

[0100] In the application, an activation layer is also connected between the above two semantic convolution layers for enhancing the image features passing through the semantic convolution layer. Specifically, referring to Figure 6The content within the dashed-line box, the activation function in the activation layer is a non-linear activation function. For example, the activation function is the ReLU6 activation function. For the semantic image features processed by the semantic convolution layer, the non-linear ReLU6 activation function can perform feature transformation on the semantic image features, enabling the obtained semantic image features to better express the important features of the third reverse image features. Then, after inputting the semantic image features processed by the activation layer into another convolutional layer, the semantic convolution module can extract more and complex features from the semantic image features. Finally, the semantic image features can no longer be processed by the activation function in the activation layer, but instead, a preset linear function is directly used to perform linear regression on the semantic image features (the image feature matrix of the semantic image features), that is Figure 6 the BN linear in it represents performing linear regression on the semantic image features, and the initial semantic information features are output. Among them, the purpose of finally using the linear function to process the semantic image features is that after inputting the third reverse image features into the above-mentioned semantic convolution layer and activation layer for processing, on the basis of realizing feature enhancement of the third reverse image features, the output semantic information features can retain more feature information in the third reverse image features.

[0101] Fuse the initial semantic information features and the third reverse image features to obtain fused features.

[0102] Perform inverse convolution processing on the fused features to obtain the semantic information features.

[0103] In applications, the fusion method of fusing the initial semantic information features and the third reverse image features is similar to the fusion method of fusing the second reverse image features and the third reverse image features. For specific explanations, refer to the above explanation in S1034, and no further elaboration will be provided here.

[0104] Specifically, refer to Figure 6 the dashed-line box in it, where Figure 6In the dashed box, P1 (the third reverse image feature) has two arrows pointing to the right. One arrow points from P1 to the semantic convolutional layer, activation layer, semantic convolutional layer, and linear regression in the dashed box in sequence, indicating that P1 is input into the semantic convolutional layer and activation layer in the semantic convolutional module for feature processing in sequence to obtain the initial semantic information feature. The other arrow points directly from P1 to the arrow after linear regression, indicating that P1 is fused with the initial semantic information feature to obtain the fused feature. After that, after performing reverse convolutional processing on the fused feature, the semantic information feature can be obtained. At this time, the arrow for obtaining the semantic information feature is divided into two arrows again starting from the dashed box. One arrow points upward and points to P2 (the second reverse image feature), which indicates that the initial target feature is obtained by performing step S10342 (fusing P1 and the semantic information feature). The other arrow points to the right and points to a preset activation function (Sigmoid), indicating that the semantic information feature is enhanced through the activation function to obtain the enhanced semantic information feature. Finally, perform step S10343: perform feature selection on the initial target feature according to the semantic information feature (the enhanced semantic information feature) to obtain the target feature P3.

[0105] In the application, performing reverse convolutional processing on the fused feature can specifically be understood as enlarging the size of the feature image of the fused feature. Specifically, reference can be made to the explanation of reverse convolutional processing in S1032 above, and no further description will be given here.

[0106] Please refer to Figure 6 , in a specific embodiment, the digital recognition network model is obtained by training the model using target training images; the target training images are obtained through the following steps S104A - S104C, which are described in detail as follows:

[0107] S104A. Obtain a digital template image in a target format.

[0108] In the application, the above digital template image can be a font file containing numbers crawled from the network in real time by the terminal device, or a font file pre - stored by the user inside the terminal device, and no limitation is made in this regard. The above target format is the storage format of the digital template image (font file) inside the terminal device. Specifically, the above target format includes but is not limited to formats such as ttf and ttc, and no limitation is made in this regard. In this embodiment, due to the lack of recognition of ttc - format fonts in existing terminal devices, it is very difficult to edit ttc fonts. Therefore, the ttf format can be used as the target format, enabling the terminal device to conveniently edit the font according to the user's processing operations.

[0109] S104B. Perform a target processing operation on the digital template image to obtain a target template image, where the target processing operation includes at least one of cropping, rotating, translating, and denoising.

[0110] In applications, the target processing operations described above include not only cropping, rotation, translation, and denoising, but may also include changing the brightness and contrast of the digital template image, without limitation. It is understood that the target processing operations performed on the digital template image may include only one of these operations or a combination of these operations.

[0111] S104C: Delete images whose similarity to the digital template image is lower than a preset threshold from the target template image to obtain the target training image.

[0112] In applications, after the digital template image undergoes the aforementioned target processing, the resulting target template image may differ significantly from the original digital template image. Under normal circumstances, if the user is unable to recognize the numbers contained in the target template image, the terminal device may calculate the similarity between the target template image and the original digital template image. If the similarity falls below a preset threshold, the target template image is deleted. The preset threshold can be set by the user based on actual circumstances.

[0113] It should be noted that digit recognition network models typically require a large amount of training data for model training. However, the amount of existing training data containing digits is far less than the required amount of training data for model training. Therefore, the terminal device can perform data augmentation processing on the existing training data containing digits through the above steps to obtain more training data. This allows the digit recognition network model trained on the target training images to achieve higher accuracy in digit recognition in target images.

[0114] See also Figure 8 , Figure 8 This is a structural block diagram of a digital recognition method device provided by an embodiment of the present application. The digital recognition method device in this embodiment includes various modules for executing Figure 1 、 Figures 3 to 5 、 Figure 7 Each step in the corresponding embodiment. Please refer to Figure 1 、 Figures 3 to 5 、 Figure 7 as well as Figure 1 、 Figures 3 to 5 、 Figure 7 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 8 The digital recognition method and apparatus 800 includes: an input module 810, a first processing module 820, a second processing module 830, and a recognition module 840, wherein:

[0115] An input module 810 for inputting an image to be recognized containing numbers into a digital recognition network model, where the digital recognition network model includes multiple convolutional layers.

[0116] A first processing module 820 for sequentially performing forward convolution processing on the image to be recognized through each convolutional layer according to the forward processing order of the multiple convolutional layers, to obtain multiple forward image features corresponding to each convolutional layer one by one. Among them, the input image of the first convolutional layer is the image to be recognized, and the input image of each remaining convolutional layer is the forward image feature output by the adjacent upper convolutional layer.

[0117] A second processing module 830 for sequentially performing reverse convolution processing on the forward image feature of the last layer through each convolutional layer according to the reverse processing order of the multiple convolutional layers, to obtain multiple reverse image features corresponding to each convolutional layer one by one. Among them, the reverse image feature of the last layer is the forward image feature of the last layer, and the input image of each remaining convolutional layer is the reverse image feature output by the adjacent upper convolutional layer.

[0118] A recognition module 840 for respectively inputting the reverse image features of the multiple convolutional layers into the classifier of the digital recognition network model to recognize the numbers contained in the image to be recognized.

[0119] In one embodiment, the second processing module 830 is further configured to:

[0120] For any convolutional layer, obtain the forward image feature of the current convolutional layer and the reverse image feature output by the adjacent upper convolutional layer to the current convolutional layer;

[0121] Perform reverse convolution processing on the forward image feature and the reverse image feature output by the upper convolutional layer to obtain the reverse image feature of the current convolutional layer.

[0122] In one embodiment, the multiple reverse image features include a first reverse image feature of the first convolutional layer; the second processing module 830 is further configured to:

[0123] Obtain a second reverse image feature of the second convolutional layer and a third reverse image feature of the third convolutional layer in the multiple convolutional layers;

[0124] Perform feature fusion on the second reverse image feature and the third reverse image feature to obtain a target feature;

[0125] Perform reverse convolution processing on the target feature and the forward image feature of the first convolutional layer to obtain the first reverse image feature of the first convolutional layer.

[0126] In one embodiment, the digital recognition network model includes a semantic convolution module; the second processing module 830 is further configured to:

[0127] Input the third reverse image feature into the semantic convolution module for feature processing to obtain the semantic information feature in the third reverse image feature;

[0128] Perform feature fusion on the second reverse image feature and the semantic information feature to obtain an initial target feature;

[0129] Perform feature selection on the initial target feature according to the semantic information feature to obtain the target feature.

[0130] In one embodiment, the second processing module 830 is further configured to:

[0131] Successively pass the third reverse image feature through the semantic convolution layer and the activation layer in the semantic convolution module for feature processing to obtain an initial semantic information feature;

[0132] Perform feature fusion on the initial semantic information feature and the third reverse image feature to obtain a fusion feature;

[0133] Perform inverse convolution processing on the fusion feature to obtain the semantic information feature.

[0134] In one embodiment, the second processing module 830 is further configured to:

[0135] Use a preset activation function to perform strengthening processing on the semantic information feature to generate the strengthened semantic information feature.

[0136] In one embodiment, the digital recognition network model is obtained by training the model using target training images, and the digital recognition method device 800 further includes the following modules for obtaining target training images;

[0137] An acquisition module, configured to acquire a digital template image in a target format.

[0138] A third processing module, configured to perform target processing operations on the digital template image to obtain a target template image, where the target processing operations include at least one of cropping, rotation, translation, and denoising.

[0139] A deletion module, configured to delete images with a similarity lower than a preset threshold from the target template image to obtain the target training image.

[0140] It should be understood that Figure 8 In the structural block diagram of the digital recognition device shown, each unit / module is used to execute Figure 1 、 Figures 3 to 5 、 Figure 7the steps in the corresponding embodiments, and for Figure 1 , Figures 3 to 5 , Figure 7 the steps in the corresponding embodiments have been explained in detail in the above embodiments. For details, please refer to Figure 1 , Figures 3 to 5 , Figure 7 and Figure 1 , Figures 3 to 5 , Figure 7 the relevant descriptions in the corresponding embodiments, which will not be elaborated here.

[0141] Figure 9 FIG. Figure 9 is a structural block diagram of a terminal device provided in another embodiment of the present application. As shown in Figure 9 , the terminal device 900 of this embodiment includes: a processor 910, a memory 920, and a computer program 930 stored in the memory 920 and executable on the processor 910, such as a program for a digital recognition method. When the processor 910 executes the computer program 930, the steps in the above-mentioned various embodiments of the digital recognition method are implemented, such as Figure 1 S101 to S104 shown in. Or, when the processor 910 executes the computer program 930, the functions of the above-mentioned Figure 8 corresponding modules in the embodiments are implemented. For example, Figure 8 the functions of modules 810 to 840 shown in. For details, please refer to Figure 8 the relevant descriptions in the corresponding embodiments.

[0142] Exemplarily, the computer program 930 can be divided into one or more units. One or more units are stored in the memory 920 and executed by the processor 910 to complete the present application. One or more units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 930 in the terminal device 900.

[0143] The terminal device may include, but is not limited to, a processor 910 and a memory 920. Those skilled in the art can understand that Figure 9 FIG. is only an example of the terminal device 900 and does not constitute a limitation on the terminal device 900. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, the terminal device may further include input / output devices, network access devices, buses, etc.

[0144] The so-called processor 910 may be a central processing unit, or may also be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0145] The memory 920 may be an internal storage unit of the terminal device 900, such as the hard disk or memory of the terminal device 900. The memory 920 may also be an external storage device of the terminal device 900, such as a plug-in hard disk, a smart memory card, a flash memory card, etc. equipped on the terminal device 900. Further, the memory 920 may also include both the internal storage unit of the terminal device 900 and the external storage device.

[0146] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included within the protection scope of the present application.

Claims

1. A digital recognition method, characterized in that, Including: Input the image to be recognized containing numbers into the digital recognition network model, where the digital recognition network model includes multiple convolutional layers and a semantic convolutional module; According to the forward processing order of the multiple convolutional layers, sequentially perform forward convolutional processing on the image to be recognized through each convolutional layer to obtain multiple forward image features corresponding to each convolutional layer one by one. Among them, the input image of the first convolutional layer is the image to be recognized, and the input image of each remaining convolutional layer is the forward image feature output by the adjacent upper convolutional layer; According to the reverse processing order of the multiple convolutional layers, sequentially perform reverse convolutional processing on the forward image feature of the last layer through each convolutional layer to obtain multiple reverse image features corresponding to each convolutional layer one by one. Among them, the reverse image feature of the last layer is the forward image feature of the last layer, and the input image of each remaining convolutional layer is the reverse image feature output by the adjacent upper convolutional layer; the multiple reverse image features include the first reverse image feature of the first convolutional layer; Input the reverse image features of the multiple convolutional layers into the classifier of the digital recognition network model respectively to recognize the numbers contained in the image to be recognized; The step of sequentially performing reverse convolutional processing on the forward image feature of the last layer through each convolutional layer according to the reverse processing order of the multiple convolutional layers to obtain multiple reverse image features corresponding to each convolutional layer one by one includes: Obtain the second reverse image feature of the second convolutional layer and the third reverse image feature of the third convolutional layer in the multiple convolutional layers; the second convolutional layer and the third convolutional layer are the convolutional layers closest to the first convolutional layer; Perform feature fusion on the second reverse image feature and the third reverse image feature to obtain a target feature; Perform reverse convolutional processing on the target feature and the forward image feature of the first convolutional layer to obtain the first reverse image feature of the first convolutional layer; The step of performing feature fusion on the second reverse image feature and the third reverse image feature to obtain a target feature includes: Perform feature processing on the third reverse image feature through the semantic convolutional layer, activation layer, and linear regression layer in the semantic convolutional module in sequence to obtain an initial semantic information feature; there are two semantic convolutional layers, the activation layer is connected between the two semantic convolutional layers, and the last semantic activation layer is connected to the linear regression layer; Perform feature fusion on the initial semantic information feature and the third reverse image feature to obtain a fusion feature; Perform reverse convolutional processing on the fusion feature to obtain the semantic information feature; Perform feature fusion on the second reverse image feature and the semantic information feature to obtain an initial target feature; Perform feature selection on the initial target feature according to the semantic information feature to obtain the target feature.

2. The digital recognition method according to claim 1, wherein The step of sequentially performing reverse convolutional processing on the forward image feature of the last layer through each convolutional layer according to the reverse processing order of the multiple convolutional layers to obtain multiple reverse image features corresponding to each convolutional layer one by one includes: For any convolutional layer, obtain the forward image features of the current convolutional layer and the backward image features output by the previous convolutional layer adjacent to the current convolutional layer; Perform backward convolution processing on the forward image features and the backward image features output by the previous convolutional layer to obtain the backward image features of the current convolutional layer.

3. The digital recognition method according to claim 1, characterized in that, Before performing feature selection on the initial target features according to the semantic information features to obtain the target features, it includes: Use a preset activation function to strengthen the semantic information features to generate the strengthened semantic information features.

4. The digital recognition method according to any one of claims 2-3, characterized in that, The digital recognition network model is obtained by training the model using target training images; the target training images are obtained through the following steps: Obtain a digital template image in a target format; Perform target processing operations on the digital template image to obtain a target template image, and the target processing operations include at least one of cropping, rotation, translation, and denoising; Delete the images with a similarity lower than a preset threshold to the digital template image from the target template image to obtain the target training images.

5. A digital recognition device, characterized in that, It includes: An input module for inputting an image to be recognized containing numbers into the digital recognition network model, and the digital recognition network model includes multiple convolutional layers and a semantic convolution module; A first processing module for sequentially performing forward convolution processing on the image to be recognized through each convolutional layer according to the forward processing order of the multiple convolutional layers to obtain multiple forward image features corresponding to each convolutional layer one by one, where the input image of the first convolutional layer is the image to be recognized, and the input image of each of the remaining convolutional layers is the forward image features output by the adjacent previous convolutional layer; A second processing module for sequentially performing backward convolution processing on the forward image features of the last layer through each convolutional layer according to the backward processing order of the multiple convolutional layers to obtain multiple backward image features corresponding to each convolutional layer one by one, where the backward image features of the last layer are the forward image features of the last layer, and the input image of each of the remaining convolutional layers is the backward image features output by the adjacent previous convolutional layer; the multiple backward image features include the first backward image features of the first convolutional layer; A recognition module for respectively inputting the backward image features of the multiple convolutional layers into the classifier of the digital recognition network model to recognize the numbers contained in the image to be recognized; The second processing module is further used for: Obtain the second backward image features of the second convolutional layer and the third backward image features of the third convolutional layer in the multiple convolutional layers; the second convolutional layer and the third convolutional layer are the convolutional layers closest to the first convolutional layer; Perform feature fusion on the second backward image features and the third backward image features to obtain target features; Perform backward convolution processing on the target features and the forward image features of the first convolutional layer to obtain the first backward image features of the first convolutional layer; The second processing module is further used for: The third reverse image feature is sequentially subjected to feature processing through the semantic convolution layer, activation layer, and linear regression layer in the semantic convolution module to obtain an initial semantic information feature; there are two semantic convolution layers, the activation layer is connected between the two semantic convolution layers, and the last semantic activation layer is connected to the linear regression layer; The initial semantic information feature and the third reverse image feature are subjected to feature fusion to obtain a fusion feature; The fusion feature is subjected to reverse convolution processing to obtain the semantic information feature; The second reverse image feature and the semantic information feature are subjected to feature fusion to obtain an initial target feature; Feature selection is performed on the initial target feature according to the semantic information feature to obtain the target feature.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 4 is implemented.

Images