Text recognition method, device, electronic device and storage medium

Abstract

The present application discloses a text recognition method, device, electronic device and storage medium, the method comprising obtaining a text image to be recognized; extracting a feature map of the text image to be recognized based on a pre-constructed text detection network, and classifying the text pixels of the fused feature map to generate at least one candidate text box, wherein the text detection network comprises a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, the second feature extraction layer upsamples the output result of the first feature extraction layer, and the third feature extraction layer downsamples the output result of the second feature extraction layer; according to a preset target text format, screening the candidate text boxes to obtain a target text box, and recognizing the target text box to obtain a recognition result. Based on this, the present application can simultaneously utilize the high-resolution information of low-level features and the high-semantic information of high-level features, so that the text pixel position detection is more accurate, and the accuracy and efficiency of text recognition are improved.

Description

Technical Field

[0001] The present application generally relates to the field of image processing technology, and specifically relates to a text recognition method, device, electronic device and storage medium. Background Art

[0002] Smart food cabinets, also known as delivery and ordering systems, are mainly used to store fast food, making people's daily lives more convenient. In actual use, smart food cabinets can verify the identity of users by pressing buttons and scanning food codes, so as to complete food pickup.

[0003] In the process of implementing the present invention, the inventors found that the accuracy of the related art in identifying the meal pickup code is low, especially when there is text at a rotated or curved angle on the meal pickup code, the error is even greater and the detection takes a long time. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the related art, it is desirable to provide a text recognition method, device, electronic device and storage medium that can improve the accuracy and efficiency of text recognition.

[0005] In a first aspect, the present application provides a text recognition method, the method comprising:

[0006] Get the text image to be recognized;

[0007] Based on a pre-built text detection network, a feature map of the to-be-recognized text image is extracted, and the fused feature map is subjected to text pixel classification to generate at least one candidate text box, wherein the text detection network comprises a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, the second feature extraction layer upsamples an output result of the first feature extraction layer, and the third feature extraction layer downsamples an output result of the second feature extraction layer;

[0008] According to a preset target text format, the candidate text boxes are screened to obtain a target text box, and the target text box is recognized to obtain a recognition result.

[0009] Optionally, in some embodiments of the present application, the method further includes:

[0010] The output result of the first feature extraction layer is subjected to residual feature enhancement processing to obtain ratio-invariant context information.

[0011] Optionally, in some embodiments of the present application, the step of filtering the candidate text boxes to obtain the target text box according to a preset target text format includes:

[0012] Obtaining the type and number of characters in the target text format;

[0013] The characters in the candidate text box are traversed, and a text box having the same character type and number as the candidate text box is determined as the target text box.

[0014] Optionally, in some embodiments of the present application, the method further includes:

[0015] detecting the aspect ratio of the candidate text box, and if the aspect ratio is different from a first preset threshold, filtering out the candidate text box; and / or,

[0016] The number of text pixels in the candidate text box is detected, and if the number of text pixels is different from a second preset threshold, the candidate text box is filtered out.

[0017] Optionally, in some embodiments of the present application, the method further includes:

[0018] Calculate the edges and corner points of the target text box;

[0019] When the edges and corners are deformed, the target text box is corrected to obtain a standard text box.

[0020] Optionally, in some embodiments of the present application, the step of identifying the target text box to obtain a recognition result further includes:

[0021] According to a preset target code format, the characters in the target text box are recognized to obtain the recognition result.

[0022] Optionally, in some embodiments of the present application, recognizing characters in the target text box according to a preset target code format to obtain the recognition result includes:

[0023] Obtaining character fonts and position order in the target code format;

[0024] The characters in the target text box are extracted according to the character font, and the characters are arranged according to the position order to obtain the recognition result.

[0025] In a second aspect, the present application provides a text recognition device, the device comprising:

[0026] An acquisition module, used for acquiring a text image to be recognized;

[0027] A generation module, configured to extract a feature map of the to-be-recognized text image based on a pre-built text detection network, and classify text pixels on the fused feature map to generate at least one candidate text frame, wherein the text detection network comprises a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, the second feature extraction layer upsamples an output result of the first feature extraction layer, and the third feature extraction layer downsamples an output result of the second feature extraction layer;

[0028] The recognition module is used to filter the candidate text boxes to obtain a target text box according to a preset target text format, and recognize the target text box to obtain a recognition result.

[0029] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores one or more programs, and the processor loads and executes the programs to implement the text recognition method described in the first aspect.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the text recognition method as described in the first aspect.

[0031] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0032] The embodiments of the present application provide a text recognition method, device, electronic device and storage medium, which extract a feature map of a text image to be recognized through a pre-constructed text detection network, wherein the second feature extraction layer of the text detection network upsamples the output result of the first feature extraction layer, and the third feature extraction layer downsamples the output result of the second feature extraction layer, thereby being able to simultaneously utilize the high-resolution information of low-level features and the high-semantic information of high-level features, making text pixel position detection more accurate and improving text recognition accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 A basic flow chart of a text recognition method provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a feature pyramid network structure provided in an embodiment of the present application;

[0036] Figure 3A schematic diagram of the structure of AugFPN provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a target code provided in an embodiment of the present application;

[0038] Figure 5 A structural diagram of a text recognition device provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the structure of another text recognition device provided in an embodiment of the present application;

[0040] Figure 7 A structural diagram of another text recognition device provided in an embodiment of the present application;

[0041] Figure 8 A structural diagram of another text recognition device provided in an embodiment of the present application;

[0042] Fig. 9 A structural schematic diagram of a text recognition device provided in another embodiment of the present application;

[0043] Fig.10 A schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the application described can be implemented in an order other than those illustrated or described herein.

[0046] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0047] For ease of understanding and explanation, the following Figures 1 to 10 The text recognition method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail.

[0048] Please refer to Figure 1 , which is a basic flow chart of a text recognition method provided in an embodiment of the present application, the text recognition method comprises the following steps:

[0049] S101, obtaining a text image to be recognized.

[0050] It should be noted that the source of the text image to be recognized in the embodiment of the present application may include but is not limited to a still image or a video dynamic image. For example, a user holds a shopping receipt and points it at the camera of an electronic device, and the image of the shopping receipt captured by the camera is a still image. For another example, a user holds a shopping receipt and passes in front of the camera of an electronic device, and the camera captures the process and extracts the image of the shopping receipt as a video dynamic image. The advantage of this setting is that it can meet a variety of application scenarios and is more convenient to use.

[0051] S102, based on the pre-built text detection network, extract the feature map of the text image to be recognized, and classify the text pixels on the fused feature map to generate at least one candidate text box.

[0052] It should be noted that the above-mentioned text detection network may include but is not limited to a feature pyramid network. The main network of the feature pyramid network is a convolutional neural network (CNN), which has the advantage of lightweight detection, thereby improving the processing efficiency of the embedded system. Optionally, the embodiment of the present application is also based on the deep neural network inference engine of MNN (MobileNeural Network) to accelerate the calculation of the convolutional neural network, and at the same time utilizes the heterogeneous computing of the NPU, GPU and NEON instruction set of the embedded system to recognize text in real time.

[0053] For example, Figure 2 As shown, the feature pyramid network (AugFPN) in the embodiment of the present application may include a first feature extraction layer (such as C2 to C5), a second feature extraction layer (such as P2 to P5) and a third feature extraction layer (such as P2' to P5' and P6). Among them, C1 corresponds to the text image to be recognized, the second feature extraction layer upsamples the output result of the first feature extraction layer with a magnification of 2 times, and the third feature extraction layer downsamples the output result of the second feature extraction layer with a magnification of 0.5 times. In addition, the dimension of P2' is 32 2 , the dimension of P3' is 642 , the dimension of P4' is 128 2 , the dimension of P5' is 256 2 , and the dimension of P6 is 512 2 .

[0054] Alternatively, if Figure 3 As shown, in the embodiment of the present application, the feature pyramid network can also perform residual feature augmentation on the output result of the first feature extraction layer to obtain context information with unchanged ratio. At the same time, after the feature map fusion, the region of interest (ROI) features are adaptively learned. The advantage of this setting is that it can narrow the semantic gap and reduce information loss, thereby greatly improving the text detection accuracy under the condition of unchanged inference speed.

[0055] Optionally, the embodiment of the present application uses a progressive scale expansion algorithm to extract similar shapes from the fused feature map, thereby achieving classification of text pixels. The advantage of this setting is that it can avoid the failure to correctly segment adjacent text during image segmentation, thereby improving the accuracy of text recognition.

[0056] S103, according to a preset target text format, filter the candidate text boxes to obtain a target text box, and recognize the target text box to obtain a recognition result.

[0057] Exemplarily, the embodiment of the present application obtains the character type and number in the target text format, and then traverses the characters in each candidate text box to determine the text box with the same character type and number as the target text box. For example, the target text format includes 5 characters, 1 of which is an English character and 4 are Arabic characters, that is, A1234.

[0058] Optionally, in the process of screening the candidate text boxes to obtain the target text box, the embodiment of the present application can also detect the aspect ratio and the number of text pixels of the candidate text box. If the aspect ratio is different from the first preset threshold, and the number of text pixels is different from the second preset threshold, the candidate text box is filtered out. Alternatively, the embodiment of the present application can also detect the aspect ratio of the candidate text box. If the aspect ratio is different from the first preset threshold, the candidate text box is filtered out. Alternatively, the embodiment of the present application can also detect the number of text pixels in the candidate text box. If the number of text pixels is different from the second preset threshold, the candidate text box is filtered out. The advantage of such a setting is that candidate text boxes that do not meet the requirements can be excluded in advance, thereby improving processing efficiency.

[0059] Optionally, after determining the target text box, the embodiment of the present application can also calculate the edges and corners of the target text box to determine whether it is a regular rectangle, so that when the edges and corners are deformed, for example, the target text box is corrected using a fast distortion correction model based on a convolutional neural network to obtain a standard text box. The advantage of this setting is that it can avoid the deformation of characters, improve the accuracy of text recognition, and further improve processing efficiency.

[0060] Optionally, when the embodiment of the present application recognizes the target text box and obtains the recognition result, it can also perform optical character recognition (OCR) on the characters in the target text box according to the preset target code format to obtain the recognition result. Exemplarily, the embodiment of the present application obtains the character font and position order in the target code format, then extracts the characters in the target text box according to the character font, and arranges the characters according to the position order to obtain the recognition result. For example, Figure 4 As shown, it is a schematic diagram of a target code provided by an embodiment of the present application. At this time, the target code can be the pick-up code on the pick-up order, and the target code format includes 5 characters, of which the first character is an uppercase English character and the last four characters are Arabic numerals, then the recognition result is A0218.

[0061] The text recognition method provided in the embodiment of the present application extracts a feature map of a text image to be recognized through a pre-constructed text detection network, wherein the second feature extraction layer of the text detection network upsamples the output result of the first feature extraction layer, and the third feature extraction layer downsamples the output result of the second feature extraction layer, thereby being able to simultaneously utilize the high-resolution information of low-level features and the high-semantic information of high-level features, thereby making text pixel position detection more accurate and improving text recognition accuracy and efficiency.

[0062] Based on the above embodiments, the present application provides a text recognition device. The text recognition device can be used for Figures 1 to 4 In the corresponding text recognition method, such as Figure 5 As shown, the text recognition device 100 includes:

[0063] An acquisition module 101 is used to acquire a text image to be recognized;

[0064] A generating module 102 is used to extract a feature map of a text image to be recognized based on a pre-built text detection network, and to classify text pixels on the fused feature map to generate at least one candidate text box;

[0065] The text detection network includes a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, the second feature extraction layer upsamples the output result of the first feature extraction layer, and the third feature extraction layer downsamples the output result of the second feature extraction layer;

[0066] The recognition module 103 is used to filter the candidate text boxes according to a preset target text format to obtain a target text box, and recognize the target text box to obtain a recognition result.

[0067] Optionally, in some embodiments of the present application, the generation module 102 is further used to perform residual feature enhancement processing on the output result of the first feature extraction layer to obtain ratio-invariant context information.

[0068] Optionally, in some embodiments of the present application, Figure 6 As shown, the identification module 103 also includes:

[0069] The acquisition unit 1031 is used to acquire the type and number of characters in the target text format;

[0070] The determination unit 1032 is used to traverse the characters in the candidate text boxes and determine a text box with the same character type and number as the target text box.

[0071] Optionally, in some embodiments of the present application, Figure 7 As shown, the identification module 103 also includes:

[0072] A filtering unit 1033 is used to detect the aspect ratio of the candidate text box, and if the aspect ratio is different from a first preset threshold, the candidate text box is filtered out; and / or,

[0073] The number of text pixels in the candidate text box is detected, and if the number of text pixels is different from a second preset threshold, the candidate text box is filtered out.

[0074] Optionally, in some embodiments of the present application, Figure 8 As shown, the identification module 103 also includes:

[0075] A calculation unit 1034, used to calculate the edge and corner points of the target text box;

[0076] The correction unit 1035 is used to correct the target text box to obtain a standard text box when there is deformation of the edges and corner points.

[0077] Optionally, in some embodiments of the present application, Fig. 9 As shown, the identification module 103 also includes:

[0078] The recognition unit 1036 is used to recognize the characters in the target text box according to a preset target code format to obtain a recognition result.

[0079] Optionally, in some embodiments of the present application, the identification unit 1036 is further configured to:

[0080] Get the character font and position order in the target code format;

[0081] The characters in the target text box are extracted according to the character font, and the characters are arranged in order of position to obtain the recognition result.

[0082] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0083] The text recognition device provided in the embodiment of the present application comprises an acquisition module for acquiring a text image to be recognized, a generation module for extracting a feature map of the text image to be recognized based on a pre-constructed text detection network, and performing text pixel classification on the fused feature map to generate at least one candidate text box; further, the recognition module is used to screen the candidate text boxes according to a preset target text format to obtain a target text box, and recognize the target text box to obtain a recognition result. The embodiment of the present application extracts a feature map of the text image to be recognized through a pre-constructed text detection network, wherein the second feature extraction layer of the text detection network upsamples the output result of the first feature extraction layer, and the third feature extraction layer downsamples the output result of the second feature extraction layer, thereby being able to simultaneously utilize the high-resolution information of the low-level features and the high-semantic information of the high-level features, making the text pixel position detection more accurate, and improving the accuracy and efficiency of text recognition.

[0084] Based on the above examples, please refer to Fig.10 , which is a basic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 200 includes one or more processors 2001 and one or more memories 2002. The memory 2002 stores at least one instruction, at least one program, code set or instruction set. The instruction, program, code set or instruction set is loaded and executed by the processor 2001 to implement the steps of the above-mentioned text recognition method. Optionally, Fig.10 The electronic device 200 shown may also include a peripheral device interface 2003, and the peripheral device interface 2003 is also connected to an input device 2004, an output device 2005, a display screen 2006, a power supply 2007, and a data acquisition device 2008, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). It should be noted that Fig.10 The components and structures of the electronic device 200 shown are merely exemplary and non-limiting. The electronic device 200 may also have other components and structures as required.

[0085] Processor 2001 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device 200 and control other components in the electronic device 200 to perform desired functions.

[0086] The memory 2002 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 2001 may run the program instructions to implement the client functions (implemented by the processing device) and / or other desired functions in the embodiment of the present application. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0087] The input device 2004 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0088] The output device 2005 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0089] The data acquisition device 2008 can acquire images of the object and store the acquired images in the memory 2002 for use by other components. Exemplarily, the data acquisition device 2008 can be a camera.

[0090] Exemplarily, the various components in the example electronic device for implementing the text recognition method according to the embodiment of the present application can be integrated or dispersed, such as integrating the processor 2001, memory 2002, input device 2004 and output device 2005 into one, while separating the data acquisition device 2008.

[0091] Exemplarily, the example electronic device for implementing the text recognition method according to the embodiment of the present application can be implemented as a terminal such as a food collection cabinet, a storage cabinet, and a delivery cabinet.

[0092] As another aspect, an embodiment of the present application provides a computer-readable storage medium for storing program code, where the program code is used to execute any one of the text recognition methods of the aforementioned embodiments.

[0093] As yet another aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the implementations of the text recognition methods in the aforementioned embodiments.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0096] In addition, each functional module in each embodiment of the present application may be integrated into a processing unit, or each module may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0097] Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the text recognition method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text recognition method, characterized in that: The method comprises: Get the text image to be recognized; Based on a pre-built text detection network, a feature map of the to-be-recognized text image is extracted, and the fused feature map is subjected to text pixel classification to generate at least one candidate text box, wherein the text detection network comprises a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, the second feature extraction layer upsamples an output result of the first feature extraction layer, and the third feature extraction layer downsamples an output result of the second feature extraction layer; Performing residual feature enhancement processing on the output result of the first feature extraction layer to obtain ratio-invariant context information; According to a preset target text format, the candidate text boxes are screened to obtain a target text box, and the target text box is recognized to obtain a recognition result.

2. The method according to claim 1, characterized in that The step of filtering the candidate text boxes to obtain the target text box according to the preset target text format includes: Obtaining the type and number of characters in the target text format; The characters in the candidate text box are traversed, and a text box having the same character type and number as the candidate text box is determined as the target text box.

3. The method according to claim 2, characterized in that The method further comprises: detecting the aspect ratio of the candidate text box, and filtering out the candidate text box if the aspect ratio is different from a first preset threshold; and / or, The number of text pixels in the candidate text box is detected, and if the number of text pixels is different from a second preset threshold, the candidate text box is filtered out.

4. The method according to any one of claims 2 or 3, characterized in that: The method further comprises: Calculate the edges and corner points of the target text box; When the edges and corners are deformed, the target text box is corrected to obtain a standard text box.

5. The method according to claim 1, characterized in that The identifying the target text box to obtain the identification result also includes: According to a preset target code format, the characters in the target text box are recognized to obtain the recognition result.

6. The method according to claim 5, characterized in that The character recognition in the target text box according to the preset target code format to obtain the recognition result includes: Obtaining the character font and position order in the target code format; The characters in the target text box are extracted according to the character font, and the characters are arranged according to the position order to obtain the recognition result.

7. A text recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a text image to be recognized; A generation module, configured to extract a feature map of the to-be-recognized text image based on a pre-built text detection network, and to perform text pixel classification on the fused feature map to generate at least one candidate text frame, wherein the text detection network comprises a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, the second feature extraction layer upsamples an output result of the first feature extraction layer, and the third feature extraction layer downsamples an output result of the second feature extraction layer; and to perform residual feature enhancement processing on an output result of the first feature extraction layer to obtain context information with a constant ratio; The recognition module is used to filter the candidate text boxes to obtain a target text box according to a preset target text format, and recognize the target text box to obtain a recognition result.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein one or more programs are stored in the memory, and the processor loads and executes the programs to implement the text recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the text recognition method according to any one of claims 1 to 6.

Images