A text recognition method and device, electronic equipment, and storage medium

Abstract

The application discloses a text recognition method and device, an electronic device and a storage medium, and the method comprises the following steps: positioning a character sequence and a starting point of the character sequence in a text image to obtain a character segmentation mask graph and a character starting point mask graph; thereby determining a corresponding polygon detection frame, and performing outer expansion on the polygon detection frame by a preset multiple to obtain a polygon point set formed by points contained in the polygon detection frame after the outer expansion; determining a starting point in the polygon point set according to the character starting point mask graph, and sequentially arranging the points in the polygon point set in a clockwise direction according to the starting point; correcting the text image according to the polygon point set after the arrangement to obtain a horizontal character picture, and performing text recognition on the horizontal character picture to obtain text contained in the text image. By means of the outer expansion on the polygon detection frame, the application solves the problem that the detection frame is pressed against a line, and meanwhile, the polygon point set formed by the outer expansion can be used to correct and recognize the curved text image, so that the recognition accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of character recognition, and particularly relates to a text recognition method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, optical character recognition (OCR) technology based on computer vision has been widely applied in various fields.

[0003] Optical character recognition technology refers to that an electronic device checks characters printed on a paper document, determines character shapes and a detection frame formed by a character shape contour through dark and bright detection, and then translates the characters in the detection frame into computer text by using a character recognition method, so that the text in a text image can be recognized. However, the optical character recognition technology can only detect horizontal text, and when the character shape obtained through dark and bright detection is irregular text, the corresponding detection frame is prone to line compression, resulting in incomplete character content in the detection frame, and thus affecting the text recognition rate. That is, it is still a challenging problem to perform character recognition on text of any shape in natural scene images / industrial scenes with diverse expressions of text images. SUMMARY

[0004] To solve the above technical problems, an embodiment of the present application provides a text recognition method and device, electronic equipment and computer readable storage medium.

[0005] According to an aspect of an embodiment of the present application, a text recognition method is provided, including: locating a character sequence in a text image and a starting point of the character sequence to obtain a character segmentation mask image and a character starting point mask image; determining a polygon detection frame corresponding to the character segmentation mask image, and performing outer expansion on the polygon detection frame by a preset multiple to obtain a polygon point set formed by points contained in the polygon detection frame after outer expansion; determining a starting point in the polygon point set according to the character starting point mask image, and sequentially sorting the points in the polygon point set in a clockwise direction according to the starting point; correcting the text image according to the polygon point set after sorting to obtain a horizontal character image, and performing text recognition on the horizontal character image to obtain text contained in the text image.

[0006] In another example embodiment, the determining the polygon bounding box corresponding to the character segmentation mask image and determining polygon parameters of the polygon bounding box comprises: determining the polygon bounding box corresponding to the character segmentation mask image, and determining polygon parameters of the polygon bounding box; and performing outer expansion on the polygon bounding box by a preset multiple, and calculating an outer expansion pixel value of the polygon bounding box according to the polygon parameters and the preset multiple; and determining a polygon point set formed by points contained in the polygon bounding box after the outer expansion according to the outer expansion pixel value.

[0007] In another example embodiment, the determining the polygon bounding box corresponding to the character segmentation mask image and determining polygon parameters of the polygon bounding box comprises: obtaining an initial point set formed by points contained in the polygon bounding box; and calculating polygon parameters of the polygon bounding box according to the initial point set.

[0008] In another example embodiment, the determining the starting point in the polygon point set according to the character starting point mask image comprises: determining a center point of the character starting point mask image; calculating Euclidean distance values between the center point and each point in the polygon point set, and screening out a point in the polygon point set corresponding to a minimum Euclidean distance value as the starting point.

[0009] In another example embodiment, the correcting the text image according to the sorted polygon point set to obtain a horizontal character picture, and performing text recognition on the horizontal character picture to obtain text contained in the text image comprises: dividing the polygon point set into a first curve point set and a second curve point set; performing Bezier curve calculation on the first curve point set and the second curve point set respectively to obtain pixel points contained in the text image; and mapping the pixel points contained in the text image to a preset output rectangle to obtain a horizontal character picture in a positive direction.

[0010] In another example embodiment, the locating the character sequence and the starting point of the character sequence in the text image obtains a character segmentation mask image and a character starting point mask image, including: locating the character sequence and the starting point of the character sequence in the text image to obtain at least two character segmentation mask images and a corresponding number of character starting point mask images; determining the positioning information of each character segmentation mask image and the positioning information of each character starting point mask image; grouping and matching the at least two character segmentation mask images and the corresponding number of character starting point mask images according to the determined positioning information to obtain at least two groups of mask image combinations, so as to recognize the text contained in the text image through the at least two groups of mask image combinations; wherein each group of mask image combinations contains a character segmentation mask image and a corresponding character starting point mask image.

[0011] In another example embodiment, the character sequence and the starting point of the character sequence in the text image are located through the FPN feature network.

[0012] According to an aspect of an embodiment of the present application, a text recognition device includes: an acquisition unit configured to locate a character sequence and a starting point of the character sequence in a text image to obtain a character segmentation mask image and a character starting point mask image; an operation unit configured to determine a polygon detection frame corresponding to the character segmentation mask image, and to perform outer expansion on the polygon detection frame by a preset multiple to obtain a polygon point set formed by points contained in the outer-expanded polygon detection frame; a preprocessing unit configured to determine a starting point in the polygon point set according to the character starting point mask image, and to perform clockwise sorting on the points in the polygon point set according to the starting point; a correction recognition unit configured to correct the text image according to the sorted polygon point set to obtain a horizontal character picture, and to perform text recognition on the horizontal character picture to obtain text contained in the text image.

[0013] According to an aspect of an embodiment of the present application, an electronic device includes: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the text recognition method as described above.

[0014] According to an aspect of an embodiment of the present application, a computer-readable storage medium has stored thereon computer-readable instructions, which, when executed by a processor of a computer, cause the computer to perform the text recognition method as described above.

[0015] In the technical solution provided in the embodiments of the present application, the character sequence in the text image and the starting point of the character sequence are accurately positioned to obtain a character segmentation mask image and a character starting point mask image and a polygon detection box corresponding to the character segmentation mask image, and the polygon detection box is expanded by a preset multiple to avoid the influence of the detection box on the character recognition accuracy, and the points in the polygon point set formed by the points included in the expanded polygon detection box are sorted clockwise to improve the data labeling quality of the polygon point set, so that the text image is corrected according to the polygon point set with high data standard quality, and the accuracy of text recognition can be further improved.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the present application, and together with the specification, serve to explain the principles of the present application. It is obvious that the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. In the drawings:

[0018] Figure 1 is a schematic diagram of an implementation environment related to the present application;

[0019] Figure 2 is a flowchart of a text recognition method according to an exemplary embodiment of the present application;

[0020] Figure 3 is Figure 2 the flowchart of step S200 in the embodiment shown in an exemplary embodiment;

[0021] Figure 4 is Figure 2 the effect diagram of the polygon detection box before and after expansion in the embodiment shown;

[0022] Figure 5 is Figure 2 the effect diagram of the character segmentation mask image and the character starting point mask image in the embodiment shown;

[0023] Figure 6 is Figure 2 the flowchart of step S400 in the embodiment shown in an exemplary embodiment;

[0024] Figure 7 is a flowchart of a text recognition method according to an exemplary embodiment of the present application;

[0025] Figure 8 is a block diagram of a text recognition apparatus according to an example embodiment of the present application;

[0026] Figure 9 A structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0027] The example embodiments will be described in detail with reference to the drawings, of which example embodiments are shown. In the following description, the same numbers are used to designate the same or similar elements unless otherwise indicated. The embodiments described in the following example embodiments do not represent all the implementations in accordance with this application. Rather, they are merely examples in accordance with some aspects of this application as detailed in the appended claims.

[0028] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the drawings are merely example illustrations, and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed depending on the actual situation.

[0030] In the present application, "a plurality of" means two or more. The association relationship of "and / or" describes the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally means that the associated objects before and after are in an "or" relationship.

[0031] First of all, artificial intelligence (AI) is to use digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0032] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other major directions.

[0033] Machine vision is a branch of artificial intelligence that is rapidly developing. Simply put, machine vision is to replace the human eye with a machine to measure and judge. The machine vision system converts the target to be taken into an image signal through machine vision products, transmits it to a dedicated image processing system, obtains the shape information of the target, and converts it into a digital signal according to the pixel distribution and brightness, color and other information; The image system performs various operations on these signals to extract the features of the target, and then controls the action of the on-site equipment according to the judgment result. Machine vision is a comprehensive technology, including image processing, mechanical engineering technology, control, electric light source lighting, optical imaging, sensors, analog and digital video technology, computer software and hardware technology.

[0034] OCR (Optical Character Recognition) is a kind of machine vision, which refers to the process of checking the printed characters on paper by electronic equipment, determining their shape by detecting light and dark patterns, and then translating them into computer text by character recognition method; That is, for printed characters, use optical methods to convert the characters in paper documents into black and white dot matrix image files, and convert the characters in the image into text format through recognition software, for further editing and processing by word processing software.

[0035] The embodiments of the present application propose a text recognition method and device, electronic equipment, and computer readable storage medium, which are mainly related to character recognition technology in machine vision included in artificial intelligence technology. The embodiments will be described in detail below.

[0036] First, please refer to Figure 1 , Figure 1 is a schematic diagram of an implementation environment related to the present application. The implementation environment includes a terminal 10 and a server 20, and the terminal 10 and the server 20 communicate through a wired or wireless network.

[0037] Server 20 performs background processing on text images requiring text recognition. By expanding the polygon detection boxes corresponding to the character sequence mask image, it solves the problem of detection boxes overlapping lines. Simultaneously, it uses the polygon point set formed by the points contained in the polygon detection boxes to correct and recognize curved text images, thereby obtaining the text contained in the text image. The obtained text is then transmitted to terminal 10 for display. Compared to existing optical character recognition solutions, the text recognition method provided in this implementation environment can improve the recognition accuracy of text of arbitrary shapes.

[0038] It should be noted that, Figure 1 In the implementation environment shown, terminal 10 can be any electronic device such as a smartphone, tablet, laptop, or computer; server 20 can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. No restrictions are imposed here.

[0039] Figure 2 This is a flowchart illustrating a text recognition method in an exemplary embodiment of this application. The method can be applied to... Figure 1 The implementation environment shown, and by Figure 1 The method is specifically executed by server 20 in the illustrated embodiment environment. However, in other implementation environments, this method can be executed by devices in those environments, and this embodiment does not impose any limitations on this.

[0040] like Figure 2 As shown, in an exemplary embodiment, the text recognition method may include steps S100 to S400, which are described in detail below:

[0041] Step S100: Locate the character sequence and the starting point of the character sequence in the text image to obtain the character segmentation mask and the character starting point mask.

[0042] After obtaining the text image containing text, the character sequence and the starting point of the character sequence in the text image are located. Then, the located character sequence and other areas in the text image other than the character sequence are segmented into polygons according to the outer contour shape of the character sequence to obtain the character segmentation mask image of the character sequence. By locating the starting point of the character sequence, the corresponding character starting point mask image on the character segmentation mask image is obtained.

[0043] The character sequence and the starting point of the character sequence in the embodiment are features extracted by the feature network, and the character sequence and the starting point of the character sequence are extracted as features in the text image, so as to determine the position information of the character sequence and the starting point of the character sequence in the text image. The method for determining the position information of the starting point of the character sequence includes but is not limited to the following described method, for example, the starting point located at the top left of the character sequence can be determined according to the writing habit of modern characters from left to right, the starting point located at the top right of the character sequence can be determined according to the writing habit of ancient characters from right to left, and the starting point located at the top of the character sequence can be determined when the character sequence is in a vertical shape.

[0044] After the character sequence and the starting point of the character sequence are extracted as features in the text image by the feature network, and the position information of the character sequence and the starting point of the character sequence in the text image is determined, the outer contour of the character sequence is used to perform polygon segmentation on the character sequence and other regions of the text image according to the shape of the outer contour of the character sequence, and rectangular segmentation is performed on the character starting point, so as to obtain a character segmentation mask image and a character starting point mask image, which are shown in Figure 3 , which are the effect schematic diagrams of the character segmentation mask image and the character starting point mask image in the embodiment, wherein A p is the character segmentation mask image, A q is the character starting point mask image.

[0045] It should be noted that the feature network mentioned above includes but is not limited to the FPN segmentation network (feature pyramid network). The FPN segmentation network locates the character sequence and the starting point of the character sequence in the text image, and obtains the character segmentation mask image and the character starting point mask image by segmentation. The FPN can perceive objects of different sizes using feature maps of different resolutions, and the output features have both bottom visual information and high-level semantic information through continuous up-sampling and cross-layer fusion mechanism. The semantic of a single low-level feature map is not rich enough and cannot be directly used for classification, while the deep-level feature is more reliable. The FPN combines lateral connection with top-down connection to obtain feature maps of different resolutions, and they all contain the semantic information of the original deepest feature map. Therefore, the FPN can independently predict the multi-scale feature maps of the input object, and the result is the pixel with the highest confidence, so that the application of the FPN segmentation network can realize accurate positioning of the text, thereby improving the accuracy of detection.

[0046] In step S200, a polygon detection frame corresponding to the character segmentation mask image is determined, and the polygon detection frame is expanded by a preset multiple to obtain a polygon point set formed by points contained in the expanded polygon detection frame.

[0047] Since the outer contour of the character segmentation mask is obtained by polygon segmentation based on the shape of the character sequence, the polygon detection box corresponding to the character segmentation mask for character recognition can be determined based on the circumscribed polygon formed by the outer contour of the character segmentation mask. After obtaining the polygon detection box, it is expanded by a preset multiple.

[0048] By expanding the polygon detection box corresponding to the character segmentation mask, and extending the original polygon detection box outward proportionally to form a new polygon detection box, the problem of detection box lines affecting character recognition accuracy can be solved. In other words, the area that the polygon detection box can detect is expanded within a suitable expansion range. After expansion, the part of the character that is pressed by the line will be located within the polygon detection box, thereby improving the accuracy of character recognition by ensuring the integrity of the character sequence.

[0049] like Figure 4 The diagram shows a comparison of the polygon detection box before and after expansion. Bi is the expanded polygon detection box, and Bn is the original polygon detection box. It is obvious that the original polygon detection box Bn has a line overlap, especially the letter "CAD" in the figure. Due to the line overlap, the letter content is severely missing. After expansion, the contact and line overlap between Bi and the character content in the box is reduced. All the English letters in the figure can be completely located within the area of ​​Bi, reducing the impact of the detection box line overlap on the recognition accuracy.

[0050] Step S300: Determine the starting point of the polygon point set based on the character starting point mask image, and sort the points in the polygon point set clockwise according to the starting point.

[0051] The character start point mask image obtained through feature network processing has a rectangular structure, such as... Figure 3 As shown, the method for determining the center point of the character start point mask is to obtain the center point of the character start point mask based on the vertex coordinates of the rectangle in which the character start point mask is located. This can be expressed as:

[0052] center(x,y)=(np.mean(x1...x4), np.mean(y1...y4)).

[0053] Here, center(x,y) represents the coordinates of the center point, xn and yn (n=1,2,3,4) represent the x and y coordinates of the vertices of the rectangle, and np.mean() represents the average value function.

[0054] After the center point and the coordinates of the center point are obtained, the Euclidean distance values between the center point and each point in the polygon point set after expansion can be calculated through the coordinates. The point in the polygon point set corresponding to the minimum Euclidean distance value is the starting point of the polygon point set sorting. After the starting point is confirmed, the points in the polygon point set are sorted clockwise according to the starting point. The clockwise sorting is as follows:

[0055] first_index=np.argsort(distEclud(center,new_box))[0];

[0056] new_box=new_box[first_index:]+new_box[:first_index];

[0057] new_box=new_box[:-1]。

[0058] Wherein, first_index represents the starting point, new_box represents the polygon point set, distEclud represents a function for calculating the Euclidean distance between the center point and each point in the polygon point set, and np.argsort represents a sorting function.

[0059] If the starting point of the character sequence is not determined, the points in the polygon point set will be sorted in disorder, which will cause distortion and error of the result when the text image is corrected according to the polygon point set. Therefore, after the starting point of the polygon point set is confirmed and the polygon point set is sorted clockwise according to the starting point according to the above steps, the polygon point set has high-quality data labeling. The text image is corrected according to the polygon point set with high-quality data labeling, which can further improve the accuracy of text recognition.

[0060] In step S400, the text image is corrected according to the sorted polygon point set to obtain a horizontal character picture, and the text contained in the text image is obtained by text recognition on the horizontal character picture.

[0061] It should be noted that the tool for correcting the text image in this embodiment can be a Bezier module. The Bezier module is highly dependent on the quality of data labeling. Therefore, the polygon point set with high-quality data labeling after sorting is input into the Bezier module. The original curved or irregular text image can be aligned and corrected to the set output rectangle through the Bezier curve, so as to obtain a horizontal character picture corresponding to the character sequence mask graph, and then the text contained in the text image is obtained by text recognition on the horizontal character picture through a text recognition network or other recognition tools, thereby improving the accuracy of text recognition.

[0062] As can be seen from the above, in the method provided in the embodiment, by expanding the polygon detection frame, the problem of line compression of the detection frame is solved, and the polygon point set formed by the points contained in the polygon detection frame is sorted, and the curved text image is corrected and recognized by using the sorted polygon point set, thereby further improving the accuracy of text recognition. The above method provided in the embodiment can be applied to text detection and recognition in natural scenes and printed / painted text recognition in industrial / traffic scenes, and can correct text of any shape, and is not limited to characters printed on paper.

[0063] Referring to Figure 5 , Figure 5 is Figure 2 the flowchart of step S200 in the embodiment. As shown in Figure 5 , step S200 can specifically include steps S201 to S203, and the polygon point set formed by the points contained in the expanded polygon detection frame is determined through the above steps, which are described in detail as follows:

[0064] Step S201, according to the character segmentation mask image, the corresponding polygon detection frame is determined, and the polygon parameters of the polygon detection frame are determined.

[0065] After the feature network segments the text image, the character segmentation mask image is obtained, which is the segmentation result. The polygon outer contour of the character segmentation mask image is the segmentation result contour, and then the polygon detection frame corresponding to the character segmentation mask image is obtained according to the segmentation result contour. Before the polygon detection frame is expanded by a preset multiple, the initial point set formed by the points contained in the polygon detection frame is calculated by a polygon fitting function (cv2.approxPolyDP), and then the polygon parameters of the polygon detection frame are calculated according to the initial point set.

[0066] For example, the polygon parameters can include polygon area and polygon perimeter, which are calculated based on the coordinates of the polygon points obtained when the feature network locates the character sequence, and therefore will not be described here. The specific calculation formula of the polygon parameters is as follows:

[0067] area = Polygon.Polygon(box).area();

[0068] peri = perimeter(box).

[0069] Wherein, area represents the polygon area, peri represents the polygon perimeter, box represents the initial point set, Polygon.Polygon().area() represents the polygon area calculation function, and perimeter represents the perimeter calculation function. In step S202, the polygon detection frame is expanded by a preset multiple, and the expanded pixel value of the polygon detection frame is calculated according to the polygon parameters and the preset multiple.

[0070] It is worth noting that the preset multiple of the expansion in the present application is a parameter that can be customized according to requirements, and the preferred value range is 1-1.5 times. After the polygon detection frame is expanded, the expanded pixel value of the polygon detection frame can be obtained according to the polygon parameters obtained in step S201 and the applied preset multiple, and the specific calculation formula is:

[0071] offset = area * (1-rate 2 ) / peri.

[0072] Wherein, offset represents the expanded pixel value, rate represents the preset multiple, and area and peri represent the polygon parameters including the polygon area and the polygon perimeter. The expanded pixel value is used to calculate the polygon point set corresponding to the points contained in the polygon detection frame after the initial point set is expanded by the preset multiple.

[0073] In step S203, the polygon point set formed by the points contained in the expanded polygon detection frame is determined according to the expanded pixel value.

[0074] It should be noted that after the expanded pixel value is determined, the polygon point set corresponding to the points contained in the expanded polygon detection frame is obtained according to the expanded pixel value, and the specific calculation formula is:

[0075] new_box = pco.Execute(-offset).

[0076] Wherein, new_box represents the polygon point set, pco.Execute represents the function of calculating the polygon point set according to the expanded pixel value, and offset represents the expanded pixel value.

[0077] According to the expanded pixel value and the initial point set, the polygon point set after expansion can be accurately obtained in the embodiment, and the feature network does not need to position the points contained in the polygon detection frame after expansion again, thereby avoiding the increase of the operation amount.

[0078] Referring to Figure 6 , Figure 6 is Figure 2 the flowchart in step S400 in the embodiment shown in FIG. 4. As Figure 5As shown, step S400 can include steps S401 to S404, and the text contained in the text image is obtained through the above steps, which are described in detail as follows:

[0079] Step S401, the polygon point set is divided into a first curve point set and a second curve point set.

[0080] It should be noted that for the division of the polygon point set, the division standard can be to divide up and down based on the plane space, and the number of points included in the first curve point set and the second curve point set is consistent, which not only ensures the quickness of the division, but also ensures the synchronous operation of the first curve point set and the second curve point set subsequently.

[0081] Step S402, respectively, the first curve point set and the second curve point set are subjected to Bezier curve calculation to obtain the pixel points contained in the text image.

[0082] It should be noted that for the Bezier curve calculation of the first curve point set and the second curve point set, the Bezier curve control points of the first curve point set and the second curve point set need to be calculated first, which is specifically as follows:

[0083] CONTROL_POINTS top =REAL_POINTS top *BezierCoeff(t) -1 ;

[0084] CONTROL_POINTS down =REAL_POINTS down *BezierCoeff(t) -1 .

[0085] Wherein, REAL_POINTS top represents the first curve point set, REAL_POINTS down represents the second curve point set, CONTROL_POINTS top represents the Bezier curve control point of the first curve point set, CONTROL_POINTS down represents the Bezier curve control point of the second curve point set, and BezierCoeff(t) -1 represents a related matrix function in the definition formula of Bezier, which is used to calculate the curve control point.

[0086] The Bezier curve control points are obtained, and the output rectangular region and the width and height of the output rectangle need to be set, and then the first curve point set and the second curve point set are controlled to perform cyclic mapping in the output rectangular region to obtain the pixel points. In this embodiment, the pixel points corresponding to the width and height are preferably 100 and 25, that is, the cycle W output= 100 pixel points W i , cycle H output = 25 pixel points H i . Wherein, W output represents the width of the output rectangle, H output represents the height of the output rectangle, W i and H i represent the pixel points corresponding to the width and height of the output rectangle respectively.

[0087] Step S403, mapping the pixel points contained in the text image to the preset output rectangle to obtain the horizontal character picture in the positive direction.

[0088] Before mapping the pixel points contained in the text image to the preset output rectangle, the corresponding coefficient of the output rectangle to the text image needs to be calculated according to the width and height of the pixel points and the output rectangle, which is as follows:

[0089] t_w = W i / W output , t_h = H i / H output .

[0090] Wherein, t_w and t_h are the corresponding coefficients of the output rectangle to the text image, W output represents the width of the output rectangle, H output represents the height of the output rectangle, W i and H i represent the pixel points corresponding to the width and height of the output rectangle respectively.

[0091] Then, the points of the first curve point set boundary t_p and the second curve point set boundary b_p are calculated using the above corresponding coefficients and the definition formula of the Bezier curve, which is as follows:

[0092] t_p = BezierCoeff(t_w) * CONTROL_POINTS top ;

[0093] b_p = BezierCoeff(1-t_w) * CONTROL_POINTS down .

[0094] Then, the sampling points op are linearly indexed by calculating the first curve point set boundary t_p and the second curve point set boundary b_p, and the calculation formula is as follows: op = bp * t_h + tp * (1-t_h).

[0095] Finally, the mapping of the pixel points contained in the text image on the preset output rectangle is completed through the above linear indexed sampling points, and a horizontal character picture in a positive direction is obtained. The positive direction herein is an adjective of a picture state, which can correspond to the position of the starting point on the character sequence or be set by the user, and is not limited herein. The mapping process can be specifically represented as:

[0096] result_img[g_ih,g_iw,∶]=org_img[int(op[0][1]),int(op[0][0]),∶]。

[0097] In step S404, text recognition is performed on the horizontal character picture to obtain the text contained in the text image.

[0098] It should be noted that the tool for text recognition of the horizontal character includes but is not limited to CRNN recognition network. CRNN stands for Convolutional Recurrent Neural Network, which is mainly used for end-to-end recognition of indefinite length text sequence. It does not need to cut single characters first, but converts text recognition into a sequence learning problem with time sequence dependence, that is, sequence recognition based on images. The application premise of CRNN is to obtain the detection content in the image, and to have a recognizable text sequence of any length. The model has the characteristics of fast speed, good performance, small model parameters, and is applied to the text recognition of the horizontal character picture obtained in the embodiment to further improve the accuracy of text recognition.

[0099] As can be seen from the above, in the method provided in the embodiment, the character sequence in the text image and the starting point of the character sequence are accurately positioned and segmented and detected by the feature network, so that the positioning is more accurate and the robustness is higher. The character segmentation mask graph and the character starting point mask graph are segmented and detected, and the polygon detection frame corresponding to the character segmentation mask graph is segmented and detected. By expanding the polygon detection frame by a preset multiple, the influence of the detection frame on the recognition rate caused by the line compression is avoided. The points of the polygon point set formed by the points contained in the expanded polygon detection frame are clockwise sorted, the problem of poor quality of the labeled data is solved, and the labeling cost is reduced. The sorted polygon point set is applied to the rectification module including the high-rectification-performance Bezier alignment rectification module to rectify and recognize the curved text image, that is, the text image in any shape (curved / horizontal) in various scenes can be rectified and recognized, and the accuracy and quality of text recognition are improved.

[0100] Please refer to Figure 7 , Figure 7 is a flowchart of a text recognition method according to another exemplary embodiment of the present application. In the text recognition method shown in Figure 7 , the text recognition method shown in Figure 2On the basis of the steps shown, step S100 includes steps S101' to step S103':

[0101] Step S101', the character sequence in the text image and the starting point of the character sequence are located, and at least two character segmentation mask graphs and the corresponding number of character starting point mask graphs are obtained.

[0102] Figure 7 An embodiment of the text recognition method shown is different from the embodiments shown in that Figure 2 The difference between the embodiments shown is that Figure 7 In the embodiment, at least two texts in the text image that need to be recognized are obtained, and then the multiple character sequences and the starting points of the character sequences located are segmented with other regions in the text image to obtain the character segmentation mask graph and the character starting point mask graph.

[0103] Step S102', the positioning information of each character segmentation mask graph and the positioning information of each character starting point mask graph are determined.

[0104] The position information of the character sequence and the starting point of the character sequence in the text image determined when the character sequence and the starting point of the character sequence in the text image are located in step S101' above is obtained, and then the positioning information of each character segmentation mask graph and the positioning information of each character starting point mask graph are determined.

[0105] Step S103', according to the determined positioning information, at least two character segmentation mask graphs and the corresponding number of character starting point mask graphs are matched in groups to obtain at least two mask graph combinations.

[0106] Because the character segmentation mask graph and the character starting point mask graph of the same character sequence are associated with the subsequent text recognition calculation, it is necessary to match multiple character segmentation mask graphs and character starting point mask graphs. Because the starting point is determined to be located at the top left of the character sequence according to the writing habit from left to right, the distance between the character segmentation mask graph and the character starting point mask graph can be determined according to the positioning information of the character segmentation mask graph and the character starting point mask graph, and the character segmentation mask graph and the character starting point mask graph with the smallest distance value are matched in groups as a group to obtain at least two mask graph combinations, each mask graph combination contains a character segmentation mask graph and a corresponding character starting point mask graph.

[0107] After grouping, the recognition of the text contained in the text image is performed according to the at least two mask graph combinations, and when two mask graph combinations are obtained, the specific implementation process is as follows:

[0108] Before the outer expansion of the polygon bounding box of each group by the preset multiple, the initial point set formed by the points contained in the polygon bounding box of each group is obtained by a polygon fitting function (cv2.approxPolyDP), and then the polygon parameters of the polygon bounding box are calculated according to the initial point set. In this exemplary embodiment of the present application, the polygon parameters include the polygon area and the polygon perimeter, so the polygon parameters of the two groups of mask image combinations are respectively:

[0109] area1 = Polygon.Polygon (box1). area ().

[0110] peri1 = perimeter (box1).

[0111] area2 = Polygon.Polygon (box2). area ().

[0112] peri2 = perimeter (box1).

[0113] Wherein, area1, area2 represent the polygon area, peri1, peri2 represent the polygon perimeter, box1, box2 represent the initial point set, and in this application, the combination of multiple groups of mask images is distinguished by subscript number. In this embodiment, all related calculation values are distinguished by subscripts 1 and 2 for the combination of two groups of mask images.

[0114] After the outer expansion of the polygon bounding box of each group by the preset multiple, the outer expansion pixel value of the polygon bounding box is calculated according to the polygon parameters and the preset multiple. The preset multiple of the outer expansion is a settable parameter according to the demand, and the preferred value range is 1-1.5 times. In this embodiment, the preset multiple of the outer expansion is set to 1.2 times, that is, rate = 1.2, so the outer expansion pixel values of the two groups of mask image combinations are respectively:

[0115] offset1 = area1 * (1-rate 2 ) / peri1;

[0116] offset2 = area2 * (1-rate 2 ) / peri2.

[0117] According to the outer expansion pixel value and the polygon point set which can be accurately expanded after the initial point set, the feature network does not need to position the points contained in the polygon bounding box again, which avoids the increase of the operation amount. Therefore, after the outer expansion pixel values offset1 and offset2 are determined, the polygon point set corresponding to the points contained in the polygon bounding box after the outer expansion is obtained according to the polygon point set, and the polygon point sets of the two groups of mask image combinations are respectively:

[0118] new_box1 = pco.Execute(-offset1) ;

[0119] new_box2 = pco.Execute(-offset2).

[0120] Before the text image is corrected according to the polygon point set, in order to avoid the sorting confusion in the polygon point set due to the uncertain starting point of the character sequence, which will cause the distortion and error of the result in the correction, the starting point in the polygon point set needs to be determined according to the character starting point mask image, and then the points in the polygon point set are sorted clockwise according to the starting point. The character starting point mask image is a rectangular structure, so the center points of the two groups of character starting point mask images can be obtained according to the vertex coordinates of the rectangle where the character starting point mask image is located:

[0121] Center1(x,y) =

[0122] (np.mean(box1_x1…box1_x4),np.mean(box1_y1…box1_y4)) ;

[0123] Center2(x,y) =

[0124] (np.mean(box2_x1…box2_x4),np.mean(box2_y1…box2_y4).

[0125] Wherein center(x,y) is the coordinate of the center point, xn,yn(n=1,2,3,4) is the horizontal and vertical coordinates of the rectangle vertex, and the prefix boxm(m is a natural number greater than 0) in this embodiment represents the same value as the rectangular vertex coordinate of different groups.

[0126] After obtaining the center point coordinate of each group of character starting point mask images, the Euclidean distance value between the center point and each point in the polygon point set after the expansion can be calculated, and the point in the polygon point set corresponding to the minimum Euclidean distance value is selected as the starting point of the sorting, and the points in the polygon point set corresponding to each group of mask image combination are sorted clockwise according to the starting point. The sorting method of the polygon point set corresponding to each group of mask image combination is the same as the method shown in the embodiment of the text image containing one text to be recognized, which will not be described here. Figure 2

[0127] Finally, the polygon point set of each group of mask image combination is divided into a first curve point set and a second curve point set, the text image is corrected to obtain a horizontal character picture, and the text in the text image is obtained by text recognition on the horizontal character picture. The correction process of the two groups of mask image combinations is the same.​

[0128] First, the Bezier curve control points of the first curve point set and the second curve point set are calculated to obtain the Bezier curve control points. At the same time, an output rectangular region needs to be set, and then the pixel points are obtained by controlling the first curve point set and the second curve point set to perform cyclic mapping in the output rectangular region; secondly, the corresponding coefficient of the set output rectangular region to the text image is calculated according to the pixel points and the width and height of the output rectangular region; and then the points of the first curve point set boundary and the second curve point set boundary are calculated by using the obtained corresponding coefficient and the definition formula of the Bezier curve; and then the first curve point set boundary and the second curve point set boundary are used to linearly index the sampling points by the calculation formula, and the mapping of the pixel points contained in the text image on the preset output rectangular region is completed by the linearly indexed sampling points, so as to obtain the horizontal character picture in the positive direction.

[0129] Finally, the text recognition is performed on the horizontal character picture by using the CRNN recognition network or other text recognition tools to obtain the text contained in the text image.

[0130] Therefore, by using the method of the embodiment, the character segmentation mask image and the character starting point mask image obtained by the application can be multiple groups, and after grouping and matching by using the positioning information, each group is subjected to text recognition, so as to obtain all the text information in the text image.

[0131] Thus, the technical scheme of the embodiment of the application can obtain multiple mask image groups by using the feature network capable of accurate positioning, and respectively expand the polygon detection frame corresponding to each mask image group, thereby solving the problem of line compression of the detection frame, sorting the polygon point set formed by the points in the polygon detection frame, and using the sorted polygon point set to correct and recognize the curved text image, thereby not only improving the labeling data quality of the polygon point set, but also applying the correction module with high correction performance to correct the text image, and further improving the accuracy of text recognition, so that the above method can be used for text images of various scenes and arbitrary shapes (curved / horizontal), and the accuracy and quality of text recognition are improved.

[0132] Figure 8 is a block diagram of a text recognition device 800 according to an example embodiment of the application.

[0133] As shown in Figure 8 , the device comprises:

[0134] The acquisition unit 801 is configured to locate a character sequence in the text image and a starting point of the character sequence, to obtain a character segmentation mask graph and a character starting point mask graph; the operation unit 802 is configured to determine a polygon detection frame corresponding to the character segmentation mask graph, and to perform outer expansion on the polygon detection frame by a preset multiple, to obtain a polygon point set formed by points contained in the polygon detection frame after the outer expansion; the preprocessing unit 803 is configured to determine a starting point in the polygon point set according to the character starting point mask graph, and to perform clockwise sorting on the points in the polygon point set according to the starting point; and the correction recognition unit 804 is configured to correct the text image according to the polygon point set after the sorting, to obtain a horizontal character picture, and to perform text recognition on the horizontal character picture to obtain the text contained in the text image.

[0135] The device applies the text recognition method provided in the present application, accurately locates a character sequence in a text image and a starting point of the character sequence, obtains a character segmentation mask graph, a character starting point mask graph, and a polygon detection frame corresponding to the character segmentation mask graph, performs outer expansion on the polygon detection frame by a preset multiple, avoids the influence of a detection frame line compression on the character recognition accuracy, performs clockwise sorting on the points of a polygon point set formed by the points contained in the polygon detection frame after the outer expansion, improves the data labeling quality of the polygon point set by the clockwise sorting, and corrects the text image according to the polygon point set with high data standard quality, to further improve the accuracy of text recognition.

[0136] Moreover, under the premise of ensuring the accuracy and quality of text recognition, the device can be applied to text detection and recognition in a natural scene and printed / painted text recognition in an industrial / traffic scene, corrects text of any shape, and is not limited to characters printed on paper.

[0137] In another exemplary embodiment, the device further comprises:

[0138] The outer expansion processing module is configured to determine a polygon detection frame corresponding to the character segmentation mask graph, and to determine polygon parameters of the polygon detection frame; to perform outer expansion on the polygon detection frame by a preset multiple, and to calculate an outer expansion pixel value of the polygon detection frame according to the polygon parameters and the preset multiple; and to determine a polygon point set formed by points contained in the polygon detection frame after the outer expansion according to the outer expansion pixel value.

[0139] The parameter calculation module is configured to obtain an initial point set formed by points contained in the polygon detection frame; and to calculate polygon parameters of the polygon detection frame according to the initial point set.

[0140] The screening module is configured to determine a center point of the character starting point mask graph, to screen a point in the polygon point set corresponding to a minimum Euclidean distance value as a starting point according to Euclidean distance values between the center point and the points in the polygon point set.

[0141] The correction module is configured to receive the polygon point set in clockwise order, divide the polygon point set into a first curve point set and a second curve point set, perform a Bezier curve calculation on the first curve point set and the second curve point set respectively to obtain pixel points contained in the text image, and map the pixel points contained in the text image to a preset output rectangle to obtain a horizontal character image in a positive direction.

[0142] In another example embodiment, the acquisition unit 801 is further configured to locate a character sequence in the text image and a starting point of the character sequence to obtain at least two character segmentation mask images and a corresponding number of character starting point mask images, determine positioning information of the character segmentation mask images and the character starting point mask images, and perform grouping matching on the character segmentation mask images and the character starting point mask images according to the positioning information to obtain at least two groups of character segmentation mask images and character starting point mask images.

[0143] It should be noted that the text recognition device provided in the above embodiments and the text recognition method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The text recognition device provided in the above embodiments can be used in actual applications, and the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0144] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the road condition refreshing method provided in each of the above embodiments.

[0145] Figure 9 The structure of the computer system of the electronic device suitable for implementing the embodiments of the present application is shown. It should be noted that, Figure 9 The computer system 1200 of the electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0146] As Figure 9As shown, the computer system 1200 includes a central processing unit (CPU) 1201 which can execute various appropriate actions and processes in accordance with programs stored in a read-only memory (ROM) 1202 or loaded from the storage section 1208 into a random access memory (RAM) 1203, such as executing the methods in the above-described embodiments. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0147] Connected to the I / O interface 1205 are an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable recording medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed into the storage section 1208 as necessary.

[0148] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable recording medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the systems of the present application are executed.

[0149] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0150] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems, which perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0151] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0152] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the road condition refreshing method as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0153] The above are only preferred embodiments of the present application, and are not used to limit the present application, and any modifications, equivalent replacements or improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A text recognition method, characterized by, The method comprises: locating a character sequence and a starting point of the character sequence in a text image to obtain a character segmentation mask image and a character starting point mask image; determining a polygon detection frame corresponding to the character segmentation mask image, and performing outer expansion on the polygon detection frame by a preset multiple to obtain a polygon point set formed by points contained in the polygon detection frame after the outer expansion; determining a starting point in the polygon point set according to the character starting point mask image, and sequentially arranging the points in the polygon point set in a clockwise direction according to the starting point; correcting the text image according to the polygon point set after the arrangement to obtain a horizontal character image, and performing text recognition on the horizontal character image to obtain text contained in the text image. The method comprises:

2. The text recognition method of claim 1, wherein, determining a polygon detection frame corresponding to the character segmentation mask image, and determining polygon parameters of the polygon detection frame; performing outer expansion on the polygon detection frame by a preset multiple, and calculating an outer expansion pixel value of the polygon detection frame according to the polygon parameters and the preset multiple; determining a polygon point set formed by points contained in the polygon detection frame after the outer expansion according to the outer expansion pixel value. The method comprises:

3. The text recognition method of claim 2, wherein, obtaining an initial point set formed by points contained in the polygon detection frame; calculating polygon parameters of the polygon detection frame according to the initial point set. The method comprises:

4. The text recognition method of claim 1, wherein, dividing the polygon point set into a first curve point set and a second curve point set; respectively performing Bezier curve calculation on the first curve point set and the second curve point set to obtain pixel points contained in the text image; mapping the pixel points contained in the text image to a preset output rectangle to obtain a horizontal character image in a positive direction. The method comprises:

5. The text recognition method of claim 1, wherein, locating a character sequence and a starting point of the character sequence in a text image to obtain at least two character segmentation mask images and a corresponding number of character starting point mask images; determining positioning information of each character segmentation mask image and positioning information of each character starting point mask image; ​ According to the determined positioning information, the at least two character segmentation mask graphs and the corresponding number of character starting point mask graphs are grouped and matched to obtain at least two groups of mask graph combinations, so as to recognize the text contained in the text image through the at least two groups of mask graph combinations. Each group of mask graph combinations contains a character segmentation mask graph and a corresponding character starting point mask graph.

6. The text recognition method of any one of claims 1-5, wherein, The character sequence in the text image and the starting point of the character sequence are positioned through the FPN feature network.

7. A text recognition apparatus characterized by comprising: The method comprises: an acquisition unit configured to position the character sequence in the text image and the starting point of the character sequence to obtain a character segmentation mask graph and a character starting point mask graph; an operation unit configured to determine a polygon detection frame corresponding to the character segmentation mask graph, and to perform outer expansion on the polygon detection frame by a preset multiple to obtain a polygon point set formed by points contained in the outer-expanded polygon detection frame; a preprocessing unit configured to determine a starting point in the polygon point set according to the character starting point mask graph, and to perform clockwise sorting on the points in the polygon point set according to the starting point; a correction recognition unit configured to correct the text image according to the sorted polygon point set to obtain a horizontal character picture, and to recognize the text contained in the text image by performing text recognition on the horizontal character picture. The determination of the starting point in the polygon point set according to the character starting point mask graph comprises: determining a center point of the character starting point mask graph; calculating the Euclidean distance values between the center point and each point in the polygon point set, and screening the point in the polygon point set corresponding to the minimum Euclidean distance value as the starting point.

8. An electronic device, comprising: The method comprises: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the text recognition method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer readable instruction is stored thereon, which, when executed by a processor of a computer, causes the computer to perform the text recognition method according to any one of claims 1-6.

Images