Text recognition method and device of self-adaptive learning reading starting point and medium

Abstract

The invention relates to a text recognition method and device for self-adaptive learning of a reading starting point and a medium, and the method comprises the steps: collecting a text image data set, carrying out the marking, and constructing a training set based on the marked text image; the annotation content comprises coordinates of all boundary points forming a text bounding box in the text image, and the coordinates of all the boundary points form a set according to a clockwise traversal sequence from the starting point; the starting point is a left upper corner point of a bounding box of a first character when the text is normally read; constructing a text content recognition model; a text boundary point detection module in the model carries out training through a constructed training set and is used for receiving a text image and then outputting boundary information of the text image, and the boundary information comprises a boundary point sequence composed of all boundary points of a text boundary; and performing text content recognition through the trained model. The text recognition method can perform text recognition without being limited by text directions and shapes, and is suitable for edge devices with limited hardware resources.

Description

Technical Field

[0001] The present invention relates to the field of text recognition, and in particular to a text recognition method, device and medium for adaptively learning a reading starting point. Background Art

[0002] With the rapid development of scene recognition algorithms and large models, scene text recognition algorithms have achieved high accuracy on benchmark data, and the latest progress shows a trend of saturation. However, existing algorithms still have problems such as poor recognition accuracy and slow recognition speed in actual scene applications. Especially in the scenario of autonomous driving, the performance of existing algorithms has been greatly reduced due to the limitations of hardware resources, the distortion of images collected by panoramic fisheye cameras, and the interference of motion noise caused by the high speed of vehicles. In order to better adapt to the resource limitations and usage scenarios of edge devices, algorithms need to be able to perform reasoning quickly and accurately to meet the needs of real-time applications. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a text recognition method, device and medium for adaptively learning a reading starting point.

[0004] The specific plan is as follows:

[0005] A text recognition method for adaptively learning a reading starting point comprises the following steps:

[0006] S1: Collect a text image dataset, annotate each text image in it, and build a training set based on the annotated text images; the annotation content includes the coordinates of all boundary points that constitute the text bounding box in the text image, and the coordinates of all boundary points are set in the order of clockwise traversal starting from the starting point; the starting point is the upper left corner of the bounding box of the first character when the text is read normally;

[0007] S2: construct a text content recognition model; the model includes a text boundary point detection module, a text area correction module and a text recognition module, wherein the text boundary point detection module is trained by the constructed training set and is used to output boundary information of the text image after receiving the text image, and the boundary information includes a boundary point sequence composed of all boundary points of the text boundary;

[0008] S3: Perform text content recognition on the text image to be recognized through the trained text content recognition model.

[0009] Furthermore, the text boundary point detection module uses a regression method to predict the coordinates of each boundary point.

[0010] Furthermore, the text region correction module is implemented by a spatial transformation network to correct the irregular text image corresponding to the boundary point sequence output by the text boundary point detection module into a regular text image.

[0011] Furthermore, the text recognition module is implemented using the RNN network.

[0012] Furthermore, the loss function during training of the text boundary point detection module adopts mean square error.

[0013] Furthermore, the loss function during text recognition module training adopts CTC loss.

[0014] A text recognition terminal device for adaptively learning a reading starting point comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in the embodiment of the present invention are implemented.

[0015] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0016] The present invention adopts the above technical solution, which can perform text recognition without being restricted by the direction and shape of the text, and is suitable for edge devices with limited hardware resources (such as various cameras (fisheye model, pinhole model, etc.)). BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a method according to a first embodiment of the present invention.

[0018] Figure 2 Shown is a schematic diagram of marking the starting point of a text image in this embodiment.

[0019] Figure 3 Shown is a schematic diagram of the network structure of the model in this embodiment. DETAILED DESCRIPTION

[0020] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0021] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0022] Embodiment 1:

[0023] The embodiment of the present invention provides a text recognition method for adaptively learning a reading starting point, such as Figure 1 As shown, the method comprises the following steps:

[0024] S1: Collect a text image dataset, annotate each text image in it, and build a training set based on the annotated text images.

[0025] Since each text boundary point is composed of multiple points, point p i The coordinates of are shown in formula (1), x i is the horizontal axis, y i is the ordinate, so the annotation content in this embodiment includes the coordinates of all boundary points that form the text boundary box in the text image. In order to reflect the reading direction of the text, in this embodiment, the coordinates of all boundary points are set in the order of clockwise traversal starting from the starting point, as shown in Formula 2. The starting point is the upper left corner of the boundary box of the first character when the text is read normally, that is, the starting point p 0 , p 0 ,p 1 ,p 2 ,…,p i In clockwise order, Figure 2 The above annotation information is used as supervision information for the subsequent training of the text boundary point detection module.

[0026] p i =(x i ,y i ) (1)

[0027] t j =[p 0 ,p 1 ,p 2 ,…,p i ] (2)

[0028] Among them, j represents the jth text in the image, t j Indicates the annotation information corresponding to the jth text in the image.

[0029] S2: Build a text content recognition model.

[0030] like Figure 3As shown, the model includes a text boundary point detection module, a text area correction module and a text recognition module which are adjacent to each other in sequence, wherein the model uses a lightweight network structure as a feature extraction encoder shared by the entire network structure. The text boundary point detection module is used to output the boundary information of the text image after receiving the text image, and the boundary information includes a boundary point sequence composed of all boundary points of the text boundary. The text boundary point detection module uses a regression method to predict the coordinates of each boundary point, which is used to represent each text shape of the input image. The number of output channels of the fully connected layer used is twice the number of boundary points. The text area correction module is implemented by a spatial transformation network to correct the irregular text image corresponding to the boundary point sequence output by the text boundary point detection module into a regular text image. The text recognition module (i.e., the text image decoder) is implemented using an RNN (Recurrent Neural Network) network structure.

[0031] The three modules of the above model are trained separately, where:

[0032] The text boundary point detection module is trained using the training set constructed in step S1. The loss function uses the mean square error (MSE) to measure the difference between the predicted coordinate value and the true coordinate value. The specific formula is shown in formula (3):

[0033]

[0034] Where n represents the number of batches, y i represents the true value of the i-th text image, Represents the predicted value of the i-th text image.

[0035] The loss function of the text recognition module uses CTC (Connectionist Temporal Classification) loss to calculate the sum of the probabilities of all possible alignments from the output sequence to the target sequence, and then take its logarithm and take the negative value as the loss. The calculation formula of CTC loss is shown in formula (4):

[0036]

[0037] Where n represents the number of batches, p i (Y|X) represents the probability of generating the target sequence Y given the input sequence X for the i-th text image.

[0038] S3: Perform text content recognition on the text image to be recognized through the trained text content recognition model.

[0039] The embodiments of the present invention solve the problem that existing text recognition algorithms are easily affected by the direction and shape of the text, resulting in recognition errors or missing characters. The scene text recognition algorithms currently on the market that are suitable for both multi-directional text and irregular text require complex data annotation and preprocessing, and the model network structure is complex and time-consuming, and are not suitable for edge device deployment. A method for efficient scene text recognition is proposed that can be achieved by using simple data annotation and end-to-end network structure design. The algorithm occupies less hardware resources and has good real-time performance, and is suitable for edge devices with limited hardware resources.

[0040] Embodiment 2:

[0041] The present invention also provides a text recognition terminal device for adaptively learning a reading starting point, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of the first embodiment of the present invention when executing the computer program.

[0042] Further, as an executable solution, the text recognition terminal device of the adaptive learning reading starting point can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The text recognition terminal device of the adaptive learning reading starting point may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the text recognition terminal device of the adaptive learning reading starting point is only an example of the text recognition terminal device of the adaptive learning reading starting point, and does not constitute a limitation on the text recognition terminal device of the adaptive learning reading starting point. It may include more or less components than the above, or a combination of certain components, or different components. For example, the text recognition terminal device of the adaptive learning reading starting point may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0043] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the text recognition terminal device of the adaptive learning reading starting point, and uses various interfaces and lines to connect various parts of the text recognition terminal device of the adaptive learning reading starting point.

[0044] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the text recognition terminal device of the adaptive learning reading starting point by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and an application required for at least one function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0045] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0046] If the module / unit integrated in the text recognition terminal device of the adaptive learning reading starting point is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0047] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A text recognition method for adaptively learning reading starting points, characterized in that: The following steps are involved: S1: Collect a text image dataset, annotate each text image in it, and build a training set based on the annotated text images; the annotation content includes the coordinates of all boundary points that constitute the text bounding box in the text image, and the coordinates of all boundary points are set in the order of clockwise traversal starting from the starting point; the starting point is the upper left corner of the bounding box of the first character when the text is read normally; S2: construct a text content recognition model; the model includes a text boundary point detection module, a text area correction module and a text recognition module, wherein the text boundary point detection module is trained by the constructed training set and is used to output boundary information of the text image after receiving the text image, and the boundary information includes a boundary point sequence composed of all boundary points of the text boundary; S3: Perform text content recognition on the text image to be recognized through the trained text content recognition model.

2. The text recognition method for adaptive learning reading starting point according to claim 1 is characterized in that: The text boundary point detection module uses regression method to predict the coordinates of each boundary point.

3. The text recognition method for adaptive learning reading starting point according to claim 1 is characterized in that: The text region correction module is implemented by a spatial transformation network to correct the irregular text image corresponding to the boundary point sequence output by the text boundary point detection module into a regular text image.

4. The text recognition method for adaptive learning reading starting point according to claim 1 is characterized in that: The text recognition module is implemented using the RNN network.

5. The text recognition method for adaptive learning reading starting point according to claim 1 is characterized in that: The loss function used during training of the text boundary point detection module is mean square error.

6. The text recognition method for adaptive learning reading starting point according to claim 1 is characterized in that: The loss function used during text recognition module training is CTC loss.

7. A text recognition terminal device capable of adaptively learning a reading starting point, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images