Optical character recognition method and device for image

Abstract

The invention discloses an optical character recognition method and device of an image, and belongs to the technical field of optical character recognition of images. The optical character recognition method for the image comprises the following steps: acquiring a to-be-recognized image; the to-be-recognized image is an image collected in an industrial scene, and optical characters are arranged on the to-be-recognized image; inputting the to-be-recognized image into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtaining a pixel feature image output by the last first common convolutional layer; inputting the pixel feature image into a plurality of depth separable convolution layers and a second common convolution layer which are sequentially connected in a lightweight model, and obtaining a semantic feature image output by the second common convolution layer; and determining a recognition result of the optical character in the to-be-recognized image based on the semantic feature image. According to the optical character recognition method for the image, the calculation complexity of feature extraction is reduced, so that the performance requirement of feature extraction on hardware equipment is reduced, and the method is effectively suitable for industrial scenes with limited resources.

Description

Technical Field

[0001] The present application belongs to the technical field of optical character recognition of images, and in particular, relates to a method and device for optical character recognition of images. Background Art

[0002] In industrial scenarios, optical character recognition technology is widely used in a variety of tasks, such as product identification, production line marking, packaging information reading, and barcode recognition. In related technologies, optical character recognition is mainly performed through traditional image processing models and deep learning models. However, the above methods have high computational complexity and require high-performance hardware support, which makes them difficult to apply to resource-constrained industrial scenarios. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes an optical character recognition method and device for images, which reduces the computational complexity of feature extraction, thereby reducing the performance requirements of feature extraction on hardware devices, and is effectively applicable to resource-constrained industrial scenarios.

[0004] In a first aspect, the present application provides a method for optical character recognition of an image, the method comprising:

[0005] Acquire an image to be identified; the image to be identified is an image collected in an industrial scene, and optical characters are set on the image to be identified;

[0006] Inputting the image to be recognized into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtaining a pixel feature image output by the last of the first common convolutional layers;

[0007] Inputting the pixel feature image into a plurality of depth-separable convolutional layers and a second common convolutional layer sequentially connected in the lightweight model, and obtaining a semantic feature image output by the second common convolutional layer;

[0008] Based on the semantic feature image, a recognition result of the optical character in the image to be recognized is determined.

[0009] According to the optical character recognition method for images of the present application, feature extraction is performed on images to be recognized including optical characters in industrial scenarios through a lightweight model. A pixel feature image is first extracted, and features in the pixel feature image are further extracted to obtain a semantic feature image. Thus, based on the semantic feature image, the optical character recognition result in the image to be recognized is determined. Feature extraction is performed through a lightweight model, and fewer parameters are used for feature extraction, thereby reducing the computational complexity of feature extraction, thereby reducing the performance requirements of feature extraction on hardware equipment, and is effectively applicable to resource-constrained industrial scenarios.

[0010] According to the optical character recognition method of the image of the present application, the multiple depth-separable convolutional layers include a first depth-separable convolutional module, a second depth-separable convolutional module, a third depth-separable convolutional module, a fourth depth-separable convolutional module, and a fifth depth-separable convolutional module connected in sequence, wherein:

[0011] The first depthwise separable convolution module includes a first depthwise separable convolution layer, and the first depthwise separable convolution layer includes: a dimension-raising depthwise separable convolution layer and a dimension-reducing depthwise separable convolution layer connected in sequence;

[0012] The second depthwise separable convolution module includes a plurality of second depthwise separable convolution layers, each of which includes: a dimension-raising matrix layer and a dimension-reducing depthwise separable convolution layer connected in sequence;

[0013] The third depthwise separable convolution module includes the dimension-increasing depthwise separable convolution layer and the dimension-reducing matrix layer;

[0014] The fourth depthwise separable convolution module comprises a plurality of fourth depthwise separable convolution layers connected in sequence, each of the fourth depthwise separable convolution layers comprises: the dimension-raising depthwise separable convolution layer and the dimension-reducing depthwise separable convolution layer connected in sequence;

[0015] The fifth depthwise separable convolution module comprises a plurality of fifth depthwise separable convolution layers connected in sequence, and each of the fifth depthwise separable convolution layers comprises a dimension-raising matrix layer and a dimension-reducing depthwise separable convolution layer connected in sequence.

[0016] According to the image optical character recognition method of the present application, the input end and the output end of the first depth-separable convolutional layer are connected through a residual structure.

[0017] According to the optical character recognition method for images of the present application, the first depth-separable convolutional layer downsamples the received input features by a target multiple in a first direction, and performs feature extraction on the downsampled input features; the first direction is the length or width direction of the image to be recognized.

[0018] According to the optical character recognition method for an image of the present application, a target first common convolution layer in the plurality of first common convolution layers downsamples the received input features by a target multiple in a first direction and a second direction, respectively, and extracts features from the downsampled input features; the first direction is the length or width direction of the image to be recognized, and the second direction is a direction different from the first direction in the length or width direction of the image to be recognized;

[0019] The target second ordinary convolution layer in the multiple first ordinary convolution layers downsamples the received input features by the target multiple in a first direction, and performs feature extraction on the input features obtained by the downsampling.

[0020] According to the optical character recognition method of an image of the present application, the step of obtaining the image to be recognized includes:

[0021] Collect initial images of industrial scenes;

[0022] Based on the original scale of the initial image, the initial image is scaled to a target size to obtain the image to be recognized.

[0023] According to the optical character recognition method for an image of the present application, the step of determining the recognition result of the optical character in the image to be recognized based on the semantic feature image includes:

[0024] The semantic feature image is processed by a long short-term memory network to determine the recognition result of the optical character in the image to be recognized.

[0025] In a second aspect, the present application provides an optical character recognition device for an image, the device comprising:

[0026] A first processing module is used to obtain an image to be identified; the image to be identified is an image collected in an industrial scene, and optical characters are set on the image to be identified;

[0027] A second processing module is used to input the image to be recognized into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtain a pixel feature image output by the last first common convolutional layer;

[0028] A third processing module is used to input the pixel feature image into a plurality of depth-separable convolutional layers and a second common convolutional layer sequentially connected in the lightweight model, and obtain a semantic feature image output by the second common convolutional layer;

[0029] The fourth processing module is used to determine the recognition result of the optical character in the image to be recognized based on the semantic feature image.

[0030] According to the optical character recognition device for images of the present application, feature extraction is performed on images to be recognized including optical characters in industrial scenarios through a lightweight model. A pixel feature image is first extracted, and features in the pixel feature image are further extracted to obtain a semantic feature image. Thus, based on the semantic feature image, the optical character recognition result in the image to be recognized is determined. Feature extraction is performed through a lightweight model, and fewer parameters are used for feature extraction, thereby reducing the computational complexity of feature extraction, thereby reducing the performance requirements of feature extraction on hardware equipment, and is effectively applicable to resource-constrained industrial scenarios.

[0031] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the optical character recognition method for an image as described in the first aspect above is implemented.

[0032] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for optical character recognition of an image as described in the first aspect above is implemented.

[0033] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the optical character recognition method for an image as described in the first aspect above.

[0034] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0035] A lightweight model is used to extract features of images to be recognized in industrial scenarios, including optical characters. The pixel feature image is first extracted, and then the features in the pixel feature image are further extracted to obtain a semantic feature image. Based on the semantic feature image, the optical character recognition result in the image to be recognized is determined. By using a lightweight model to extract features, fewer parameters are used for feature extraction, and the computational complexity of feature extraction is reduced, thereby reducing the performance requirements of feature extraction on hardware equipment, which is effectively applicable to resource-constrained industrial scenarios.

[0036] Furthermore, feature extraction is performed by setting multiple sequentially connected depthwise separable convolution modules. During the feature extraction process, the image is subjected to dimensionality-upgraded feature extraction to extract richer features, and the dimension of the extracted feature image is reduced to focus on more important information. The convolutional layers that have less impact on the recognition of photovoltaic characters are pruned. On the basis of ensuring the accuracy of feature extraction in industrial scenarios, the inference time of feature extraction is reduced and the efficiency of feature extraction is improved.

[0037] Furthermore, the input and output of the first depth-wise separable convolutional layer are connected through a residual structure, so that the input features can be directly transmitted to the output, which helps to retain more original features and reduce information loss during the propagation process. The residual structure can also improve the convergence speed.

[0038] Furthermore, by selecting a target first ordinary convolution layer from multiple first ordinary convolution layers, downsampling in the first direction and the second direction is performed, and a target second ordinary convolution layer is selected from multiple ordinary convolution layers, downsampling in the first direction is performed, and the size of the input features is reduced through multiple first ordinary convolution layers to reduce the computational complexity of pixel feature extraction and improve the extraction efficiency of subsequent input features.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1 It is one of the flowcharts of the optical character recognition method of an image provided in the embodiment of the present application;

[0042] Figure 2 This is the second flow chart of the optical character recognition method for an image provided in an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the structure of an optical character recognition device for an image provided in an embodiment of the present application;

[0044] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0046] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0047] In conjunction with the accompanying drawings, the optical character recognition method for an image, the optical character recognition device for an image, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0048] The optical character recognition method of an image may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0049] The terminal includes but is not limited to portable communication devices such as mobile phones or tablet computers. It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer.

[0050] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0051] The optical character recognition method for images provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the optical character recognition method for images. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The optical character recognition method for images provided in the embodiments of the present application is described below using an electronic device as an example of the execution subject.

[0052] like Figure 1 As shown, the optical character recognition method of the image includes: step 110, step 120, step 130 and step 140.

[0053] Step 110, obtaining an image to be recognized; the image to be recognized is an image collected in an industrial scene, and optical characters are set on the image to be recognized;

[0054] In this step, the industrial scene refers to the scene involving production and manufacturing equipment and products.

[0055] Industrial scenarios include but are not limited to: production line tracking and management scenarios, automated inspection scenarios, sorting scenarios, logistics scenarios, warehouse management scenarios, and equipment maintenance and maintenance scenarios.

[0056] It should be noted that industrial scenarios include scenarios in different fields, such as automobile production and manufacturing scenarios, electronic equipment production and manufacturing scenarios, and daily necessities production and manufacturing scenarios.

[0057] The optical character is a character corresponding to information related to the object to be recognized in the image to be recognized.

[0058] The information related to the object to be identified includes, but is not limited to: product identification, production line mark, packaging information and barcode identification, production date and precautions.

[0059] Optical characters include, but are not limited to, numbers, letters, and fonts corresponding to different countries.

[0060] For example, the optical character set on the image to be recognized may be a barcode mark, such as a string of numbers 123456.

[0061] It should be noted that in industrial scenarios, it may be necessary to determine information such as the serial number, batch number, and production date of the equipment and products in the industrial scenario. By collecting images of the location where the information to be identified is located, automatic identification can be performed to determine the relevant information of the equipment or product.

[0062] During the actual implementation process, the image to be recognized can be collected by an image sensor.

[0063] Taking the need to conduct quality inspection on manufactured products as an example, the image sensor can be used to capture images of the positions that need to be inspected by the inspection device. For example, to detect the production date on a food packaging bag, the image sensor can be used to capture images of the positions corresponding to the production date, or the entire image of the packaging bag can be captured by video to obtain the image to be identified.

[0064] Of course, in the actual implementation process, the image to be recognized can also be obtained by any other feasible method.

[0065] Step 120: input the image to be recognized into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtain a pixel feature image output by the last first common convolutional layer;

[0066] In this step, a lightweight model is a model with a small number of model parameters, a small amount of calculation, and a small memory footprint.

[0067] The lightweight model can be Mobilenetv4, ShuffleNet, Xception and other models.

[0068] The specific selection of the lightweight model can be based on user customization or determined based on actual conditions, and this application does not limit it.

[0069] The lightweight model can be trained using images to be identified in industrial scenarios of various different production and manufacturing categories as samples and feature images corresponding to the images to be identified as sample labels.

[0070] The lightweight model can also be pre-trained using a public simulated optical character recognition dataset as a training sample, and then trained with a dataset of industrial scenarios in a variety of different production and manufacturing categories at a smaller learning rate to obtain a lightweight model suitable for industrial scenarios, thereby improving the recognition capability of the lightweight model in industrial scenarios.

[0071] The first common convolutional layer is a convolutional layer that extracts features of the image to be recognized through a convolution kernel of a fixed size.

[0072] The number of first common convolutional layers can be customized by the user based on the accuracy requirements of feature extraction, or determined based on actual conditions. For example, the number of first common convolutional layers can be 4 or 7, which is not limited in this application.

[0073] like Figure 2 As shown in the figure, taking the number of the first ordinary convolutional layers as 5 as an example, the first ordinary convolutional layer can perform the initial stage of processing on the image to be recognized, that is, corresponding to the stem stage, and then the features output by the first ordinary convolutional layer are subjected to the first stage of feature extraction through the second ordinary convolutional layer and the third ordinary convolutional layer, and the features extracted in the first stage are subjected to the second stage of feature extraction through the fourth ordinary convolutional layer and the fifth ordinary convolutional layer to obtain a pixel feature image.

[0074] The size of the convolution kernel of the first common convolution layer can be user-defined or determined based on actual conditions. For example, the size of the convolution kernel of the first common convolution layer can be 3×3 or 5×5, which is not limited in this application.

[0075] The convolution step size of the first common convolution layer can be user-defined or determined based on actual conditions. For example, the convolution step size of the first common convolution layer is set to 2 or 4, which is not limited in this application.

[0076] Taking the convolution step size of the first ordinary convolution layer as 2 as an example, the size of the feature map output by each first ordinary convolution layer will be reduced to half of the previous size.

[0077] The pixel feature image is an image of pixel-level features such as edges, textures, and contours obtained by extracting features from the image to be identified.

[0078] Taking the letter "L" as an example, the boundary between the letter "L" and the background of the image to be recognized can be extracted, including the boundary direction, boundary strength and boundary curvature of the letter "L", as well as the line thickness and adhesion of the letter "L" in the image to be recognized.

[0079] In the actual implementation process, the image to be identified can be input into the lightweight model, and the image to be identified can be processed layer by layer through multiple first ordinary convolution layers connected in sequence. The first first ordinary convolution layer receives the image to be identified, performs the first feature extraction on the image to be identified, and then inputs the extracted features into the first ordinary convolution layer of the next layer for further feature extraction, and so on, until the pixel feature image output by the first ordinary convolution layer of the last layer is obtained. It can be understood that the features extracted by each layer are more accurate than the features extracted by the previous layer.

[0080] Step 130: input the pixel feature image into a plurality of depth-separable convolutional layers and a second common convolutional layer sequentially connected in the lightweight model, and obtain a semantic feature image output by the second common convolutional layer;

[0081] In this step, the depthwise separable convolutional layer is a convolutional layer that decomposes the convolution operation into two steps: depthwise convolution and pointwise convolution.

[0082] The specific number of depth-wise separable convolutional layers can be customized based on the accuracy requirements of feature extraction, or determined based on actual conditions. For example, the number of depth-wise separable convolutional layers can be 8 or 10, etc., which is not limited in this application.

[0083] The second common convolutional layer is a convolutional layer with a fixed-size convolutional kernel connected to the output of the depthwise separable convolutional layer in the lightweight model.

[0084] The semantic feature image is a feature image of the overall structure and semantic information of the focused character obtained by further extracting features from the pixel feature image.

[0085] Taking the optical characters in the image to be recognized as English words as an example, the actual meaning of the word can be determined by extracting the combination and context information between the letters.

[0086] In the actual execution process, after obtaining the pixel feature image, the pixel feature image can be input into multiple first ordinary convolutional layers connected in sequence, and the pixel feature image can be processed layer by layer. The first depthwise separable convolutional layer receives the pixel feature image, performs the first feature extraction on the pixel feature image, and then inputs the extracted features into the depthwise separable convolutional layer of the next layer for further feature extraction, and so on, until the features extracted by the depthwise separable convolutional layer of the last layer are obtained, and the features are further processed through the second ordinary convolutional layer to obtain the semantic feature image output by the second ordinary convolutional layer.

[0087] Step 140: Determine the recognition result of the optical character in the image to be recognized based on the semantic feature image.

[0088] In this step, the recognition result is the result obtained by performing character recognition on the semantic feature image.

[0089] In the actual implementation process, the semantic feature image can be input into a recurrent neural network model or a classification model to obtain the recognition result of the optical characters in the image to be recognized.

[0090] Of course, in the actual implementation process, the semantic feature image can also be recognized by any other feasible method to obtain the recognition result of the optical character in the image to be recognized, and this application does not limit it.

[0091] In some embodiments, step 140 may further include:

[0092] The semantic feature image is processed by a long short-term memory network to determine the recognition result of the optical character in the image to be recognized.

[0093] In this embodiment, after obtaining the semantic feature image, the semantic feature image can be input into the long short-term memory network, and based on the predefined total number of recognized characters, the recognition probability of each character is output to obtain the recognition result of the optical character in the image to be recognized.

[0094] Assuming that the total number of recognized characters is 96 and the size of the semantic feature image is 1×80, the long short-term memory network will output 1×80×96, which represents the probability of each character in the 80 character positions of a single image. That is, in the output result of the maximum supported 80-bit characters, empty characters and adjacent characters are filtered out to obtain the final characters.

[0095] According to the optical character recognition method for images provided in the embodiments of the present application, semantic feature images are recognized through long-short time memory networks, and optical characters in the image to be recognized are effectively obtained, thereby meeting the user's usage needs and improving the user experience.

[0096] The inventors found during the research and development process that in the related technologies, optical character recognition is mainly performed through traditional image processing models and deep learning models. However, the above methods have high computational complexity and require high-performance hardware support.

[0097] The present application replaces the traditional convolutional neural network with a lightweight model with fewer parameters to perform feature extraction and reduce the computational complexity of feature extraction. That is, a preliminary feature extraction is performed on the image to be identified in the industrial scenario through multiple first ordinary convolutional layers connected in sequence in the lightweight model to obtain a pixel feature image; then a deeper feature extraction is performed on the pixel feature image through multiple depth-separable convolutional layers and second ordinary convolutional layers in the lightweight model to obtain a semantic feature image. The pixel feature image is first extracted, and then the semantic feature image is extracted, so as to gradually reduce the dimension and redundant information of the data, combine the lightweight model with optical character recognition in the industrial scenario, blur the features that are not required for feature extraction, focus on the features with strong correlation, improve the extraction accuracy of the features with strong correlation, improve the extraction efficiency, have a good feature extraction effect, and can reduce the performance requirements of the feature extraction on the hardware equipment.

[0098] According to the optical character recognition method for images provided in the embodiments of the present application, feature extraction is performed on images to be recognized including optical characters in industrial scenarios through a lightweight model. A pixel feature image is first extracted, and then features in the pixel feature image are further extracted to obtain a semantic feature image. Thus, based on the semantic feature image, the optical character recognition result in the image to be recognized is determined. Feature extraction is performed through a lightweight model, and fewer parameters are used for feature extraction, thereby reducing the computational complexity of feature extraction, thereby reducing the performance requirements of feature extraction on hardware equipment, and is effectively applicable to resource-constrained industrial scenarios.

[0099] In some embodiments, step 110 may further include:

[0100] Collect initial images of industrial scenes;

[0101] Based on the original scale of the initial image, the initial image is scaled to the target size to obtain the image to be recognized.

[0102] In this step, the initial image is an image obtained by photographing the optical character to be recognized.

[0103] The original ratio is the ratio of the original image length to width.

[0104] For example, the native ratio may be 4:3, 3:2, 16:9, 1:1, etc.

[0105] The target size is the preset size of the image processed by the lightweight network.

[0106] The specific value of the target size can be user-defined or determined based on actual conditions. For example, the target size can be 32×320, or 48×480, etc., which is not limited in this application.

[0107] In the actual implementation process, an initial image in an industrial scene can be collected through an image sensor or a mobile terminal. When the size of the collected initial image is different from the target size, the initial image is scaled to the target size based on the original ratio of the initial image.

[0108] Taking the target size of 32×320 and the initial image size of 24×240 as an example, the width of the initial image can be enlarged to 32 pixels based on the original ratio of the size of the initial image, that is, 1:10. The original ratio of the initial image is the same as the ratio of the target size. When the width of the initial image is enlarged to 32 pixels, the length of the initial image will be enlarged to 320 pixels.

[0109] Taking the target size of 32×320 and the size of the initial image of 24×270 as an example, the width of the initial image can be enlarged to 32 pixels based on the original ratio of the size of the initial image, that is, 1:10. The original ratio of the initial image is different from the ratio of the target size. When the width of the initial image is enlarged to 32 pixels, the length of the initial image will be enlarged to 360 pixels, and then the pixel image with a length exceeding 320 will be cropped.

[0110] Similarly, when the initial image needs to be reduced and the width or length of the scaled image does not meet the target size, padding can be performed on the length or width so that the reduced image is of the target size to obtain the image to be recognized.

[0111] According to the optical character recognition method for images provided in the embodiments of the present application, an initial image is acquired and, based on the original ratio of the initial image, the initial image is scaled to a target size, thereby effectively processing the initial image to the same size, thereby improving the efficiency of subsequent lightweight models in processing images to be recognized.

[0112] The following is an explanation using the lightweight model Mobilenetv4.

[0113] like Figure 2 As shown, in some embodiments, the plurality of depth-wise separable convolutional layers may include a first depth-wise separable convolutional module, a second depth-wise separable convolutional module, a third depth-wise separable convolutional module, a fourth depth-wise separable convolutional module, and a fifth depth-wise separable convolutional module connected sequentially, wherein:

[0114] The first depthwise separable convolution module includes a first depthwise separable convolution layer, and the first depthwise separable convolution layer includes: a dimension-increasing depthwise separable convolution layer and a dimension-reducing depthwise separable convolution layer connected in sequence;

[0115] The second depthwise separable convolution module includes a plurality of second depthwise separable convolution layers, each of which includes: a dimension-raising matrix layer and a dimension-reducing depthwise separable convolution layer connected in sequence;

[0116] The third depthwise separable convolution module includes a dimension-increasing depthwise separable convolution layer and a dimension-reducing matrix layer;

[0117] The fourth depthwise separable convolution module includes a plurality of fourth depthwise separable convolution layers connected in sequence, each of which includes: a dimension-raising depthwise separable convolution layer and a dimension-reducing depthwise separable convolution layer connected in sequence;

[0118] The fifth depthwise separable convolution module includes a plurality of fifth depthwise separable convolution layers connected in sequence, and each fifth depthwise separable convolution layer includes a dimension-increasing matrix layer and a dimension-reducing depthwise separable convolution layer connected in sequence.

[0119] In this embodiment, the first depthwise separable convolution module is a module for extracting features from a pixel feature image.

[0120] The second depth-wise separable convolution module is a module for extracting features output by the first depth-wise separable convolution module.

[0121] The third depth-wise separable convolution module is a module for extracting features output by the second depth-wise separable convolution module.

[0122] The fourth depth-wise separable convolution module is a module for extracting features output by the third depth-wise separable convolution module.

[0123] The fifth depth-wise separable convolution module is a module for extracting features output by the fourth depth-wise separable convolution module.

[0124] The first depthwise separable convolutional layer first increases the number of feature map channels for extracting features through a dimensionality-increasing depthwise separable convolutional layer, and then reduces the number of feature map channels for extracting features through a dimensionality-reducing depthwise separable convolutional layer.

[0125] Continue to refer Figure 2 , the first depth-wise separable convolutional layer can be ExtraDWConv.

[0126] The dimension-enhanced depthwise separable convolutional layer is a convolutional layer that enhances the feature extraction capability.

[0127] The dimension-enhanced depthwise separable convolutional layer includes: depthwise convolutional layer and pointwise convolutional layer.

[0128] The dimension reduction depth separable convolutional layer is a convolutional layer that reduces the complexity and computation of feature extraction.

[0129] The dimension reduction depthwise separable convolutional layer includes: depthwise convolutional layer and pointwise convolutional layer.

[0130] The number of output channels of the point-by-point convolutional layer of the dimensionality reduction depthwise separable convolutional layer is less than the number of input channels to achieve a reduction in the number of feature channels.

[0131] In actual implementation, the size of the convolution kernel of the first depthwise separable convolutional layer can be set to 5×5 and the padding is set to 2.

[0132] Of course, the size of the convolution kernel of the first depth-separable convolution layer can also be customized by the user or determined based on actual conditions. For example, the size of the convolution kernel of the first depth-separable convolution layer is set to 3×3 or 7×7, which is not limited in this application.

[0133] It should be noted that, while ensuring that the size of the input feature image is the same as the size of the output feature image, the size of the convolution kernel and the padding size can be combined in a variety of ways.

[0134] Taking the pixel feature image with 64 channels as an example, the first depthwise separable convolution module can expand the number of channels of the pixel feature image from 60 to 120 through a dimensionality-increasing depthwise separable convolution layer, and then reduce the number of channels from 120 to 80 through a dimensionality-reducing depthwise separable convolution layer, thereby performing convolution in a higher dimension, capturing richer feature information, and outputting features with controllable feature dimensions.

[0135] The second depth-wise separable convolutional layer first increases the number of feature map channels for extracting features through a dimension-increasing matrix layer, and then reduces the number of feature map channels for extracting features through a dimension-reducing depth-wise separable convolutional layer.

[0136] The dimension-increasing matrix layer is a convolutional layer that increases the number of channels through a fully connected layer or a 1×1 convolutional layer.

[0137] The dimension reduction matrix layer is a convolutional layer that maps high-dimensional data to low-dimensional space through a transformation matrix to achieve data dimension reduction.

[0138] like Figure 2 As shown, after the first depth-wise separable convolution module, three second depth-wise separable convolution layers can be connected in sequence to obtain a second depth-wise separable convolution module.

[0139] During the actual execution process, the number of second depth-separable convolutional layers can also be customized based on the actual feature extraction requirements, for example, 2 or 5 second depth-separable convolutional layers can be set, which is not limited in this application.

[0140] The third depth-wise separable convolutional layer first increases the number of feature map channels for extracting features through a dimensionality-increasing depth-wise separable convolutional layer, and then reduces the number of feature map channels for extracting features through a dimensionality-reducing matrix layer.

[0141] like Figure 2 As shown, the first depth-wise separable convolution module, the second depth-wise separable convolution module, and the third depth-wise separable convolution module connected sequentially can be used as the third stage of feature extraction to perform feature extraction.

[0142] The fourth depth-wise separable convolution layer in the fourth depth-wise separable convolution module may be a convolution layer having the same structure as the first depth-wise separable convolution layer, and processes features output by the third depth-wise separable convolution module.

[0143] The number of fourth depth-wise separable convolutional layers can be user-defined or determined based on actual conditions. For example, the number of fourth depth-wise separable convolutional layers can be 2 or 4, which is not limited in this application.

[0144] like Figure 2 As shown, taking the number of fourth depth-wise separable convolutional layers as 2 as an example, the size of the convolution kernel of the fourth depth-wise separable convolutional layer of the second layer can be set to 5×5, and the padding can be set to 2.

[0145] The fifth depth-wise separable convolution layer in the fifth depth-wise separable convolution module may be a convolution layer having the same structure as the second depth-wise separable convolution layer, and processes features output by the fourth depth-wise separable convolution module.

[0146] The number of the fifth depth-wise separable convolutional layers can be user-defined or determined based on actual conditions. For example, the number of the fifth depth-wise separable convolutional layers can be 3 or 5, which is not limited in this application.

[0147] like Figure 2 As shown, taking the number of fifth depth-wise separable convolutional layers as 3 as an example, the size of the convolution kernel of the fifth depth-wise separable convolutional layer of the first layer can be set to 5×5, and the padding can be set to 2.

[0148] Continue to refer Figure 2 , the fourth depth-wise separable convolution module and the fifth depth-wise separable convolution module connected sequentially can be used as the fourth stage of feature extraction to perform feature extraction.

[0149] The feature extraction in the fourth stage may also include feature extraction performed by the second common convolutional layer, thereby obtaining a semantic feature image.

[0150] According to the optical character recognition method for images provided in the embodiments of the present application, feature extraction is performed by setting a plurality of sequentially connected depthwise separable convolution modules. During the feature extraction process, the image is subjected to dimensionality-upgraded feature extraction to extract richer features, and the dimension of the extracted feature image is reduced to focus on more important information. The convolutional layers that have less influence on the recognition of photovoltaic characters are pruned. On the basis of ensuring the accuracy of feature extraction in industrial scenarios, the inference time of feature extraction is reduced and the efficiency of feature extraction is improved.

[0151] In some embodiments, the input and output of the first depthwise separable convolutional layer are connected via a residual structure.

[0152] In this embodiment, the residual structure is a structure that enables the first depthwise separable convolutional layer to directly transfer input features to the output end.

[0153] It should be noted that the structure of the fourth depth-wise separable convolutional layer is the same as that of the first depth-wise separable convolutional layer. During the temporal execution process, the input and output ends of the fourth depth-wise separable convolutional layer can also be connected through a residual structure.

[0154] During the execution process, a skip connection may be added between the input and output ends of the first depthwise separable convolutional layer, so that the first depthwise separable convolutional layer can directly transfer input features to the output end structure.

[0155] According to the optical character recognition method for images provided in an embodiment of the present application, the input end and the output end of the first depth-separable convolutional layer are connected through a residual structure, so that the input features can be directly transmitted to the output end, which helps to retain more original features and reduce the loss of information during the propagation process. The residual structure can also improve the convergence speed.

[0156] In some embodiments, the first depthwise separable convolutional layer downsamples the received input features by a target multiple in a first direction, and performs feature extraction on the downsampled input features; the first direction is the length or width direction of the image to be identified.

[0157] In this embodiment, the target multiple is a preset downsampling multiple.

[0158] The specific value of the target multiple can be user-defined or determined based on actual conditions. For example, the target multiple can be 2 times or 3 times, which is not limited in this application.

[0159] Taking the input feature size received by the first depthwise separable convolutional layer as 16×160, and the first direction being the width direction of the image to be recognized as an example, when the target sampling multiple is 2, the input feature is downsampled by 2 times to obtain an 8×160 feature. After downsampling the input feature, the input feature can be further feature extracted to obtain the feature extraction result of the first depthwise separable convolutional layer.

[0160] According to the optical character recognition method for an image provided by an embodiment of the present application, by downsampling the input features by a target multiple in the first direction on a first depth-separable convolutional layer, the size of the input features in the first direction is effectively reduced, the amount of computational effort for feature extraction of the input features is reduced, and the efficiency of feature extraction is improved.

[0161] In some embodiments, a target first common convolution layer in the plurality of first common convolution layers downsamples the received input features by a target multiple in the first direction and the second direction, respectively, and performs feature extraction on the input features obtained by the downsampling;

[0162] A target second common convolution layer among the plurality of first common convolution layers downsamples the received input features by a target multiple in a first direction, and performs feature extraction on the input features obtained by the downsampling.

[0163] In this embodiment, the target first common convolutional layer is any convolutional layer among multiple first common convolutional layers.

[0164] The first direction is the length or width direction of the image to be recognized.

[0165] The second direction is a direction in the length or width direction of the image to be recognized that is different from the first direction.

[0166] When the first direction is the length direction, the second direction is the width direction.

[0167] When the first direction is the width direction, the second direction is the length direction.

[0168] The target second common convolution layer is any two common convolution layers among the multiple first common convolution layers except the target first common convolution layer.

[0169] Continue to refer Figure 2 In the actual execution process, the target first ordinary convolution layer can be the first ordinary convolution layer, that is, the ordinary convolution layer corresponding to the stem stage, and the received input features are downsampled by 2 times in the length direction and the width direction respectively in the stem stage.

[0170] The target second ordinary convolution may be the second ordinary convolution layer and the fourth ordinary convolution layer in the plurality of ordinary convolution layers.

[0171] The second and fourth ordinary convolutional layers downsample the received input features by a factor of 2 in the width direction respectively.

[0172] According to the optical character recognition method for an image provided by an embodiment of the present application, a target first ordinary convolution layer is selected from multiple first ordinary convolution layers, downsampling is performed in the first direction and the second direction, a target second ordinary convolution layer is selected from multiple ordinary convolution layers, downsampling is performed in the first direction, and the size of the input feature is reduced through multiple first ordinary convolution layers to reduce the computational complexity of pixel feature extraction and improve the extraction efficiency of subsequent input features.

[0173] In some embodiments, before step 110, the method may further include:

[0174] The image to be recognized is downsampled by a target multiple in a first direction and a second direction respectively to obtain a downsampled recognition image.

[0175] In this embodiment, the downsampled recognition image is an image obtained by downsampling the image to be recognized.

[0176] In the actual implementation process, the maximum pooling layer can be used to downsample the image to be recognized by a target multiple in the first direction and the second direction respectively to obtain a downsampled recognition image.

[0177] Continue to refer Figure 2 In the actual execution process, in the lightweight model, there are 2 convolutional layers that downsample in the first direction and the second direction respectively, and 3 convolutional layers that downsample in the first direction. If the target multiple is 2 and the size of the image to be recognized is 32x320, after downsampling of 5 convolutional layers, the size of the semantic feature image finally output by the lightweight model is 1 / 32 of the size of the image to be recognized in the first direction and 1 / 4 of the size of the image to be recognized in the second direction, that is, the size of the semantic feature image is 1×80.

[0178] According to the optical character recognition method for images provided in the embodiments of the present application, by downsampling the image to be recognized by a target multiple in the first direction and the second direction respectively, the size of the object to be recognized is effectively reduced, the amount of calculation for feature extraction of the image to be recognized is reduced, and the efficiency of feature extraction of the image to be recognized is improved.

[0179] The optical character recognition method for images provided in the embodiment of the present application can be performed by an optical character recognition device for images. In the embodiment of the present application, the optical character recognition device for images performing the optical character recognition method for images is taken as an example to illustrate the optical character recognition device for images provided in the embodiment of the present application.

[0180] The embodiment of the present application also provides an optical character recognition device for images.

[0181] like Figure 3 As shown, the optical character recognition device for an image includes: a first processing module 310 , a second processing module 320 , a third processing module 330 and a fourth processing module 340 .

[0182] The first processing module 310 is used to obtain an image to be recognized; the image to be recognized is an image collected in an industrial scene, and optical characters are set on the image to be recognized;

[0183] The second processing module 320 is used to input the image to be recognized into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtain a pixel feature image output by the last first common convolutional layer;

[0184] The third processing module 330 is used to input the pixel feature image into a plurality of depth-separable convolutional layers and a second common convolutional layer sequentially connected in the lightweight model, and obtain a semantic feature image output by the second common convolutional layer;

[0185] The fourth processing module 340 is used to determine the recognition result of the optical character in the image to be recognized based on the semantic feature image.

[0186] According to the optical character recognition device for images provided in the embodiments of the present application, feature extraction is performed on images to be recognized including optical characters in industrial scenarios through a lightweight model. A pixel feature image is first extracted, and features in the pixel feature image are further extracted to obtain a semantic feature image. Thus, based on the semantic feature image, the optical character recognition result in the image to be recognized is determined. Feature extraction is performed through a lightweight model, and fewer parameters are used for feature extraction, thereby reducing the computational complexity of feature extraction, thereby reducing the performance requirements of feature extraction on hardware equipment, and is effectively applicable to resource-constrained industrial scenarios.

[0187] In some embodiments, the first processing module 310 may also be used to:

[0188] Collect initial images of industrial scenes;

[0189] Based on the original scale of the initial image, the initial image is scaled to the target size to obtain the image to be recognized.

[0190] In some embodiments, the fourth processing module 340 may also be used to:

[0191] The semantic feature image is processed by a long short-term memory network to determine the recognition result of the optical character in the image to be recognized.

[0192] The optical character recognition device of the image in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0193] The optical character recognition device for images in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0194] The optical character recognition device for images provided in the embodiment of the present application can realize Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0195] In some embodiments, Figure 4 As shown, the embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned image optical character recognition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0196] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0197] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned image optical character recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0198] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0199] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned optical character recognition method for images when executed by a processor.

[0200] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0201] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image optical character recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0202] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0203] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0204] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0205] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0206] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0207] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for optical character recognition of an image, characterized in that: include: Obtain an image to be recognized; The image to be identified is an image collected in an industrial scene, and optical characters are set on the image to be identified; Inputting the image to be recognized into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtaining a pixel feature image output by the last of the first common convolutional layers; Inputting the pixel feature image into a plurality of depth-separable convolutional layers and a second common convolutional layer sequentially connected in the lightweight model, and obtaining a semantic feature image output by the second common convolutional layer; Based on the semantic feature image, a recognition result of the optical character in the image to be recognized is determined.

2. The method for optical character recognition of an image according to claim 1, characterized in that: The plurality of depth-wise separable convolutional layers include a first depth-wise separable convolutional module, a second depth-wise separable convolutional module, a third depth-wise separable convolutional module, a fourth depth-wise separable convolutional module, and a fifth depth-wise separable convolutional module connected in sequence, wherein: The first depthwise separable convolution module includes a first depthwise separable convolution layer, and the first depthwise separable convolution layer includes: a dimension-raising depthwise separable convolution layer and a dimension-reducing depthwise separable convolution layer connected in sequence; The second depthwise separable convolution module includes a plurality of second depthwise separable convolution layers, each of which includes: a dimension-raising matrix layer and a dimension-reducing depthwise separable convolution layer connected in sequence; The third depthwise separable convolution module includes the dimension-increasing depthwise separable convolution layer and the dimension-reducing matrix layer; The fourth depthwise separable convolution module comprises a plurality of fourth depthwise separable convolution layers connected in sequence, each of the fourth depthwise separable convolution layers comprises: the dimension-raising depthwise separable convolution layer and the dimension-reducing depthwise separable convolution layer connected in sequence; The fifth depthwise separable convolution module comprises a plurality of fifth depthwise separable convolution layers connected in sequence, and each of the fifth depthwise separable convolution layers comprises a dimension-raising matrix layer and a dimension-reducing depthwise separable convolution layer connected in sequence.

3. The method for optical character recognition of an image according to claim 2, characterized in that: An input end and an output end of the first depthwise separable convolutional layer are connected via a residual structure.

4. The method for optical character recognition of an image according to claim 2, characterized in that: The first depthwise separable convolutional layer downsamples the received input features by a target multiple in a first direction, and performs feature extraction on the downsampled input features; the first direction is the length or width direction of the image to be identified.

5. The method for optical character recognition of an image according to any one of claims 1 to 4, characterized in that: A target first common convolution layer among the multiple first common convolution layers downsamples the received input features by a target multiple in a first direction and a second direction, respectively, and extracts features from the downsampled input features; the first direction is the length or width direction of the image to be identified, and the second direction is a direction different from the first direction in the length or width direction of the image to be identified; The target second ordinary convolution layer in the multiple first ordinary convolution layers downsamples the received input features by the target multiple in a first direction, and performs feature extraction on the input features obtained by the downsampling.

6. The method for optical character recognition of an image according to any one of claims 1 to 4, characterized in that: The step of obtaining the image to be identified comprises: Collect initial images of industrial scenes; Based on the original scale of the initial image, the initial image is scaled to a target size to obtain the image to be recognized.

7. The method for optical character recognition of an image according to any one of claims 1 to 4, characterized in that: The step of determining the recognition result of the optical character in the image to be recognized based on the semantic feature image includes: The semantic feature image is processed by a long short-term memory network to determine the recognition result of the optical character in the image to be recognized.

8. An optical character recognition device for an image, characterized in that: include: A first processing module, used for acquiring an image to be identified; The image to be identified is an image collected in an industrial scene, and optical characters are set on the image to be identified; A second processing module is used to input the image to be recognized into a plurality of first common convolutional layers connected in sequence in the lightweight model, and obtain a pixel feature image output by the last first common convolutional layer; A third processing module is used to input the pixel feature image into a plurality of depth-separable convolutional layers and a second common convolutional layer sequentially connected in the lightweight model, and obtain a semantic feature image output by the second common convolutional layer; The fourth processing module is used to determine the recognition result of the optical character in the image to be recognized based on the semantic feature image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the optical character recognition method for an image as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for optical character recognition of an image as claimed in any one of claims 1 to 7 is implemented.

Images