Text recognition method, device, electronic device and readable storage medium

Abstract

The embodiments of the present application provide a text recognition method, device, electronic device, and readable storage medium, and the method relates to artificial intelligence technology and the cloud field. The method includes: obtaining a text image to be recognized, wherein the text image to be recognized includes a curved text area; extracting a text feature map of the text image to be recognized; determining the text area to be recognized and the region boundary information of the text area to be recognized in the text image to be recognized based on the text feature map, wherein the text area to be recognized includes a curved text area; correcting the text area to be recognized according to the region boundary information to obtain a corrected rectangular text area; performing text recognition on the rectangular text area to obtain a text recognition result of the text image to be recognized. In the embodiments of the present application, the curved text area can be corrected to a rectangular text area, which reduces the background noise in the text area and improves the recognition effect.

Description

Technical Field

[0001] The present application relates to the fields of image processing, artificial intelligence, and big data technology. Specifically, the present application relates to a text recognition method, device, electronic device, and readable storage medium. Background Art

[0002] Optical Character Recognition (OCR) is a key research topic in the field of computer vision, with curved text recognition being a key application. In recent years, with the development of deep learning technology, deep learning-based OCR detection and recognition has become a mainstream trend.

[0003] OCR technology consists of two networks: a detection network that locates the location of text and a recognition network that identifies its content. Existing recognition algorithms for curved text detection and recognition often focus on a single task: locating the location of the curved text. However, due to the excessive background noise inherent in curved text, the subsequent recognition network is unable to extract the text, resulting in poor recognition results. Summary of the Invention

[0004] Embodiments of the present application provide a text recognition method, apparatus, electronic device, and readable storage medium, which can improve the text recognition effect for curved text in an image.

[0005] In one aspect, an embodiment of the present application provides a text recognition method, the method comprising:

[0006] Acquire a text image to be recognized, wherein the text image to be recognized includes a curved text area;

[0007] Extracting a text feature map of the text image to be recognized;

[0008] Determine, based on the text feature map, a text region to be recognized in the text image to be recognized and region boundary information of the text region to be recognized, wherein the text region to be recognized includes a curved text region;

[0009] Correct the text area to be recognized according to the area boundary information to obtain a corrected rectangular text area;

[0010] Perform text recognition on the rectangular text area to obtain the text recognition result of the text image to be recognized.

[0011] On the other hand, an embodiment of the present application provides a text recognition device, which includes:

[0012] An image acquisition module is used to acquire a text image to be recognized, wherein the image to be processed includes a curved text area;

[0013] A feature extraction module is used to extract a text feature map of the text image to be recognized;

[0014] A boundary information determination module is used to determine a text region to be identified in an image and region boundary information of the text region to be identified based on a text feature map, wherein the text region to be identified includes a curved text region;

[0015] A region correction module is used to correct the text region to be recognized according to the region boundary information to obtain a corrected rectangular text region;

[0016] The text recognition module is used to perform text recognition on the rectangular text area and obtain the text recognition result of the text image to be recognized.

[0017] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory: the memory is configured to store a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned text recognition method.

[0018] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program runs on a computer, the computer can execute the above-mentioned text recognition method.

[0019] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0020] In an embodiment of the present application, when performing text recognition on a text image to be recognized, the text area to be recognized in the text image to be recognized and the area boundary information of the text area to be recognized can be determined, and then the text area to be recognized can be corrected based on the obtained area boundary information to obtain a corrected rectangular text area, and text recognition is performed on the corrected rectangular text area to obtain a text recognition result. At this time, for the curved text area in the text image to be recognized, since the curved text area can be corrected to a rectangular text area, the background noise in the text area can be effectively reduced. Accordingly, when performing text recognition on the text area, the recognition effect can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0022] Figure 1 A flowchart of a text recognition method provided in an embodiment of the present application;

[0023] Figure 2a A schematic diagram of a curved text area provided in an embodiment of the present application;

[0024] Figure 2b A schematic diagram of a rectangular text area provided in an embodiment of the present application;

[0025] Figure 3 Schematic diagram of boundary information provided in the embodiment of the present application;

[0026] Figure 4a A schematic diagram of a sample image provided in an embodiment of the present application;

[0027] Figure 4b A schematic diagram of a text area and a background area in a sample image provided in an embodiment of the present application;

[0028] Figure 4c A schematic diagram of the boundary of a curved text area in a sample image provided in an embodiment of the present application;

[0029] Figure 4d A schematic diagram of labeling boundary category labels for each pixel in a sample image provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the network structure of a regional prediction model provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the principle of correcting a text area to be recognized provided in an embodiment of the present application;

[0032] Figure 7 A comparative diagram of text recognition results provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the structure of a text recognition device provided in an embodiment of the present application;

[0034] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0036] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0037] The embodiments of the present application provide a text recognition method, device, electronic device, and readable storage medium, which specifically relate to computer vision technology in artificial intelligence technology.

[0038] Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0039] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0041] Optionally, the data processing / computation involved in the embodiments of the present application can be achieved based on cloud computing. Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides resources is called a "cloud." The resources in the "cloud" appear to users to be infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis. As a provider of cloud computing infrastructure capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established. Various types of virtual resources will be deployed in the resource pool for external customers to choose from. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. According to logical functional division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases and web containers. SaaS is a variety of business software, such as web portals and SMS mass senders. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0042] First, several terms involved in this application are introduced and explained:

[0043] CCL (Connected Component Analysis / Labeling): Connected domain analysis refers to finding independent connected domains in an image and marking them. A connected domain of an image is a region consisting of pixels with the same pixel value and adjacent positions. Extracting different connected domains from an image and connected domain analysis are commonly used methods in image processing, such as segmenting and identifying regions of interest in license plate recognition, text recognition, target detection, and other fields. Generally, a connected domain contains only one pixel value. Therefore, to prevent the influence of pixel value fluctuations on the extraction of different connected domains, connected domain analysis often processes binarized images.

[0044] OCR refers to the process of analyzing and processing scanned text files to extract text and layout information. Key metrics for measuring OCR system performance include rejection rate, false positive rate, recognition speed, user interface friendliness, product stability, usability, and feasibility. Debugging and utilizing auxiliary information to improve recognition accuracy are crucial OCR issues.

[0045] SGD (tochastic Gradient Descent): is a simple but very efficient method, mainly used for discriminative learning of linear classifiers under convex loss functions.

[0046] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0047] Optionally, the method provided in the embodiments of the present application can be executed by a server, a terminal device, or the interaction between a server and a terminal device. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a laptop computer, a desktop computer, a smartwatch, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0048] Figure 1 FIG. 1 shows a flow chart of a text recognition method provided in an embodiment of the present application. Figure 1 As shown, the method may include:

[0049] Step S101 : obtaining a text image to be recognized, wherein the text image to be recognized includes a curved text area.

[0050] The "text image to be recognized" refers to an image for which text recognition is required, and the image to be processed includes a curved text region. Optionally, the text image to be recognized can be a single image or a frame image in a video. The source of the text image to be recognized is not limited in this embodiment of the application.

[0051] Optionally, when the method is implemented by a terminal device, the text image to be identified can be an image captured by the terminal device based on its own image acquisition device, or it can be a saved image obtained by the terminal device from its own storage medium; when the method is implemented by a server, the text image to be identified can be an image received by the server from the terminal device.

[0052] Step S102: extracting a text feature map of the text image to be recognized.

[0053] Optionally, a text feature map of the text image to be recognized may be extracted, and text recognition may be performed on the image to be recognized based on the extracted text feature map to obtain a corresponding text recognition result.

[0054] Wherein, when extracting the text feature map of the text image to be identified, it can be implemented based on a neural network model, and the specific network structure of the neural network model is not limited in the embodiment of the present application. As an optional embodiment, the neural network model may include CNN (Convolutional Neural Networks) and FPN (Feature Pyramid Networks, Feature Pyramid Networks), and when extracting the text feature map of the text image to be identified based on the neural network model, the CNN (such as res50) network can first perform downsampling feature extraction on the text image to be identified of size 1024*1024 (i.e., 1024 pixels*1024 pixels), and obtain a 512-layer feature map of size 64*64 (i.e., 64 pixels*64 pixels); further, the feature map obtained based on the CNN network can be input to the FPN network, and the FPN network performs a set number of upsampling feature extractions, outputs a 512-layer feature map of size 128*128 (i.e., 128 pixels*128 pixels), and serves as the text feature map of the text image to be identified.

[0055] Step S103 : determining a text region to be recognized in the text image to be recognized and region boundary information of the text region to be recognized based on the text feature map, wherein the text region to be recognized includes a curved text region.

[0056] The "unrecognized text region" in the unrecognized text image refers to the region corresponding to the text to be recognized in the image, and the region boundary information of the unrecognized text region refers to the information of the boundaries that constitute the unrecognized text region. For example, if the unrecognized text region is a rectangular region, the region boundary information of the unrecognized text region may refer to the information of the top, bottom, left, and right boundaries that constitute the unrecognized text region.

[0057] Optionally, when extracting the text feature map of the text image to be recognized, the text region to be recognized in the text image to be recognized and region boundary information of the text region to be recognized may be obtained based on the text feature map.

[0058] Step S104 : Correcting the text region to be recognized according to the region boundary information to obtain a corrected rectangular text region.

[0059] Optionally, when the region boundary information of the text region to be recognized is obtained, the specific position of the text region to be recognized in the text image to be recognized is known. At this time, the text region to be recognized can be corrected based on the obtained region boundary information to obtain a corrected rectangular text region. For example, a curved text region in a text image to be recognized is as follows: Figure 2a As shown, at this time, the curved text area can be corrected into a rectangular text area based on the obtained area boundary information of the curved text area, specifically as follows Figure 2b shown.

[0060] The specific implementation method for correcting the unrecognized text region based on the region boundary information to obtain the corrected rectangular text region can be pre-configured and is not limited in the embodiments of the present application. For example, the unrecognized text region can be corrected by using a Bezier cutout method based on the obtained region boundary information to obtain the corrected rectangular text region.

[0061] Step S105 , performing text recognition on the rectangular text area to obtain a text recognition result of the text image to be recognized.

[0062] The specific implementation method for performing text recognition on the rectangular text area can be pre-configured and is not limited in the embodiments of the present application. As an optional method, text recognition can be performed on the text in the rectangular text area using OCR to obtain a text recognition result of the text image to be recognized.

[0063] It can be understood that when the text image to be recognized includes multiple text areas to be recognized, each text area to be recognized and the area boundary information of each text area to be recognized can be determined based on the text feature map, and then each text area to be recognized can be corrected and text recognition can be performed. It is also possible to determine the specified text area to be recognized in the text image to be recognized and the area boundary information of the specified text area to be recognized based on the text feature map, and then correct the specified text area to be recognized and text recognition can be performed. The embodiments of the present application are not limited to this.

[0064] In an embodiment of the present application, when performing text recognition on a text image to be recognized, the text area to be recognized in the text image to be recognized and the area boundary information of the text area to be recognized can be determined, and then the text area to be recognized can be corrected based on the obtained area boundary information to obtain a corrected rectangular text area, and text recognition is performed on the corrected rectangular text area to obtain a text recognition result. At this time, for the curved text area in the text image to be recognized, since the curved text area can be corrected to a rectangular text area, the background noise in the text area can be effectively reduced. Accordingly, when performing text recognition on the text area, the recognition effect can be effectively improved.

[0065] In an optional embodiment of the present application, the region boundary information includes boundary information of a first side of the text region to be recognized, wherein the first side is two sides consistent with the writing direction of the text in the text image to be recognized;

[0066] Correct the text area to be recognized based on the area boundary information, including:

[0067] Determining boundary information of a second side of the text area to be recognized based on the boundary information of the first side and the text area to be recognized, wherein the second side is two sides in a direction perpendicular to the writing direction;

[0068] The text area to be recognized is corrected according to the boundary information of the first side and the boundary information of the second side.

[0069] The text area to be recognized includes text to be recognized. In this case, the two sides that are consistent with the writing direction of the text can be used as the first side of the text area to be recognized, and the two sides that are perpendicular to the writing direction of the text can be used as the second side of the text area to be recognized. Figure 3 As shown, assuming that the text in the text image to be recognized is "Hello!", and the writing direction of the text is from left to right, the left boundary information and the right boundary information of the text area to be recognized can be used as the boundary information of the first side of the text area to be recognized, and the upper boundary information and the lower boundary information of the text area to be recognized can be used as the boundary information of the second side of the text area to be recognized.

[0070] Optionally, when obtaining a text feature map of the text image to be recognized, the text area to be recognized and the boundary information of the first side of the text area to be recognized can be determined based on the text feature map, and then the boundary information of the second side of the text area to be recognized can be determined based on the determined boundary information of the first side and the text area to be recognized; accordingly, the text area to be recognized can be corrected into a rectangular area based on the boundary information of each side of the text area to be recognized to reduce the background noise in the text area to be recognized, thereby improving the text recognition effect when performing text recognition subsequently.

[0071] Optionally, the specific implementation method of correcting the text area to be recognized into a rectangular area based on the boundary information on each side of the text area to be recognized can be pre-configured according to actual needs. For example, the text area to be recognized can be corrected into a rectangular area by using Belset cutout.

[0072] In an optional embodiment of the present application, determining the text region to be recognized in the text image to be recognized and the region boundary information of the text region to be recognized based on the text feature map includes:

[0073] Based on the text feature map, determine the text pixels in the text image to be recognized and the boundary pixels of the text pixels belonging to the text area to be recognized;

[0074] Determine the text area to be recognized in the text image to be recognized based on the text pixels;

[0075] Based on the boundary pixels, the region boundary information of the text image to be recognized is determined.

[0076] Among them, text pixel points refer to pixel points in the text image to be identified that are within the text area to be identified. These pixel points are used to display the text in the text image to be identified. Boundary pixel points refer to pixel points at the boundary of the text area to be identified. It can be understood that text pixel points include boundary pixel points.

[0077] Optionally, when obtaining a text feature map, it is possible to determine which pixel points in the text image to be identified are within the text area to be identified (i.e., determine the text pixel points in the text image to be identified), and which pixel points in the text image to be identified are located at the boundary of the text area to be identified (i.e., determine the boundary pixel points in the text image to be identified) based on the determined text feature map; accordingly, when it is known which ones are within the text area to be identified, the text pixel points determined at this time constitute the text area to be identified in the text image to be identified, and based on the determined boundary pixel points, the area boundary information of the text image to be identified is determined.

[0078] In an optional embodiment of the present application, the region boundary information includes boundary information of a first side of the to-be-recognized text region, where the first side is two sides consistent with a writing direction of the text in the to-be-recognized text image, and the boundary pixel points include boundary pixel points on one side of the first side and boundary pixel points on the other side of the first side;

[0079] Based on the boundary pixels, the region boundary information of the text image to be recognized is determined, including:

[0080] Determine a first boundary connected domain on one side according to boundary pixel points on one side of the first side, and determine boundary information on one side of the first side based on the first boundary connected domain;

[0081] A second boundary connected domain on the other side is determined according to boundary pixel points on the other side of the first side, and boundary information of the other side of the first side is determined based on the second boundary connected domain.

[0082] Optionally, in an embodiment of the present application, the boundary information of the first side of the text area to be identified refers to the two sides consistent with the writing direction of the text in the text image to be identified. At this time, the determined boundary pixel points may include the boundary pixel points on one side of the first side and the boundary pixel points on the other side of the first side; accordingly, a first boundary connected domain including the boundary information of one side of the first side can be obtained based on the boundary pixel points on one side of the first side, and a second boundary connected domain including the boundary information of the other side of the first side can be obtained based on the boundary pixel points on the other side of the first side, and then the boundary information of the first side of the text area to be identified can be obtained based on the determined first boundary connected domain and the second boundary connected domain; further, the boundary information of the second side of the text area to be identified can be obtained based on the determined text area to be identified and the boundary information of the first side of the text area to be identified.

[0083] Among them, the first boundary connected domain and the second boundary connected domain can be determined based on the boundary pixel points through the CCL method, and the implementation method of determining the boundary information of the first side of the text area to be identified based on the determined first boundary connected domain and second boundary connected domain is related to which areas' pixels are specifically used as boundary pixel points during pre-training. For example, during pre-training, the left boundary and the right boundary of the text area to be identified are used as the two sides of the first side of the text area to be identified, and a triangular area is constructed with the left boundary as the base, and the pixels within the triangular area are used as the boundary pixel points of one side of the first side, and a triangular area is constructed with the right boundary as the base, and the pixels within the triangular area are used as the boundary pixel points of the other side of the first side; accordingly, in actual applications, when performing text recognition on the text image to be identified, the first boundary connected domain and the second boundary connected domain determined based on the boundary pixel points are triangles, and at this time, the bases of the two triangles can be used as the boundary information of the first side of the identified text area.

[0084] In an optional embodiment of the present application, determining the boundary information of the second side of the text region to be recognized based on the boundary information of the first side of the text region to be recognized and the text region to be recognized includes:

[0085] Determining boundary key points of a second side of the text area to be recognized based on the boundary information of the first side and the text area to be recognized;

[0086] Boundary information of a second side of the to-be-recognized text region is determined based on the boundary key points and the boundary information of the first side.

[0087] Among them, boundary key points refer to some of the points used to constitute the boundary line, and which specific points in the boundary are used as boundary key points can be pre-configured, and the embodiments of this application do not limit it. For example, the starting point, end point and middle point of the boundary line can be used as boundary key points, etc. In order to better reflect the position of the boundary key points, the boundary key points can be represented by coordinates.

[0088] Optionally, in an embodiment of the present application, when it is known which pixels in the text image to be identified are within the text area to be identified, the text area to be identified in the text image to be identified can be determined by CCL. At this time, the specific presentation form of the text area to be identified can be a polygon, and it is composed of a series of coordinate points, such as (x0, y0, x1, y1,…, xn, yn). Furthermore, the boundary key points of the second side of the text area to be identified can be determined based on the first boundary information of the first side and the text area to be identified, that is, the boundary key points of the second side of the text area to be identified are determined from the coordinate points used to represent the text area to be identified, and then the determined boundary key points can be connected to obtain a line, and the line can be used as the boundary information of the second side of the text area to be identified.

[0089] In an optional embodiment of the present application, determining the boundary information of the second side of the text region to be recognized based on the boundary key points and the boundary information of the first side includes:

[0090] Determining initial boundary information of a second side of the text area to be recognized based on the boundary key points and the boundary information of the first side;

[0091] The initial boundary information is subjected to curve smoothing processing to obtain boundary information of the second side of the text area to be recognized.

[0092] Optionally, after obtaining each boundary key point, if the boundary information of the second side of the text area to be identified is obtained directly based on each boundary key point, the boundary information of the second side obtained at this time is not very accurate. At this time, the initial boundary information of the second side of the text area to be identified can be determined based on the boundary key point and the boundary information of the first side, and then the initial boundary information of the second side is subjected to curve smoothing processing to obtain the boundary information of the second side of the text area to be identified. The specific implementation method of the curve smoothing processing can be pre-configured and is not limited in the embodiment of the present application. For example, the initial boundary information can be fitted using a Bessel method to obtain the boundary information of the second side of the text area to be identified.

[0093] In an optional embodiment of the present application, determining the text region to be recognized in the text image to be recognized and the region boundary information of the text region to be recognized based on the text feature map includes:

[0094] Based on the text image to be recognized, a text feature map is extracted through the feature extraction module of the region prediction model;

[0095] Based on the text feature map, the region prediction module of the region prediction model is used to obtain the text region to be recognized in the text image to be recognized and the region boundary information of the text region to be recognized;

[0096] Among them, the regional prediction model is obtained through the following methods:

[0097] Obtaining an initial neural network model and a sample dataset, the sample dataset including each sample image and a text category label and a boundary category label for each sample image, the text category label including a labeling result of a region category in the sample image, the boundary category label including a labeling result of region boundary information of a text region in the sample image, the text region in the sample image including a curved text region, and the region category including a text region and a background region;

[0098] The initial neural network model is trained based on the sample data set until the total loss function corresponding to the initial neural network model converges, thereby obtaining a regional prediction model;

[0099] Among them, the total loss function includes a first loss function and a second loss function. The value of the first loss function represents the difference between the annotation results of the region boundary information of the text area in each sample image and the predicted region boundary information output by the model. The second loss function represents the difference between the annotation results of the region category in each sample image and the predicted results of the region category output by the model.

[0100] Optionally, the acquired text image to be recognized can be input into a region prediction model, and the feature extraction module included in the region prediction model can perform feature extraction on the text image to be recognized to obtain a text feature map, and then the obtained text feature map is input into a region prediction module included in the region prediction model. The region prediction module can obtain the text area to be recognized in the text image to be recognized and the region boundary information of the text area to be recognized based on the input text feature map.

[0101] The region prediction model can be obtained by training the initial neural network model using the acquired sample dataset. The sample dataset includes each sample image and its text category label and boundary category label. The text category label includes the annotation results of the region category in the sample image, i.e., the text region and background region in the sample image are labeled. The boundary category label includes the annotation results of the region boundary information of the text region in the sample image, i.e., the actual region boundary information of the text region in the sample image is labeled. The sample text region includes a curved text region.

[0102] Furthermore, each sample image can be input into the initial neural network model, and the initial neural network model can output the predicted text area to be identified and the predicted area boundary information of the text area to be identified corresponding to each sample image, and obtain the value of the first loss function based on the annotation results of the area boundary information of the text area in each sample image and the predicted area boundary information, and obtain the value of the second loss function based on the annotation results of the area category in each sample image and the predicted text area to be identified, and then obtain the value of the total loss function based on the value of the first loss function and the value of the second loss function, and judge whether the value of the total loss function converges. If not, the model parameters of the initial neural network model can be adjusted, and each sample image can be input into the adjusted initial neural network model again, and the value of the total loss function can be calculated again based on the output of the initial neural network model, until the training is terminated when the value of the total loss function converges, and the trained initial neural network model is used as the area prediction model.

[0103] Optionally, when training the initial neural network model based on sample data, an open source deep framework (pytorch) can be used for training. For example, the initial learning rate can be set to 0.01, and the learning rate descent method can be set to poly (a learning rate descent method, polynomial decay), the gradient update method can be SGD, and the number of training iterations can be set to 120 epochs (where 1 epoch means that all sample images are input into the initial neural network model to complete a forward calculation and back propagation process) to train the initial neural network model.

[0104] In an optional embodiment of the present application, for any sample image, the text category label includes the text category label of each pixel in the sample image, the text category label of the pixel is a background label or a text label, and the boundary category label includes the boundary category label of each pixel in the sample image, the boundary category label of the pixel is a boundary label or a non-boundary label;

[0105] The input of the initial neural network model is each sample image, and the output of the initial neural network model is the text category prediction result and boundary category prediction result of each pixel point in each sample image;

[0106] The value of the first loss function represents the difference between the boundary category label and the boundary category prediction result of each pixel point in each sample image, and the value of the second loss function represents the difference between the text category label and the text category prediction result of each pixel point in each sample image.

[0107] Among them, the text category label includes the text category label of each pixel in the sample image, and the text category label of the pixel includes one of a background label or a text label, the background label indicates that the pixel is not located in the text area to be identified, and the text label indicates that the pixel is located in the text area to be identified; the boundary category label includes the boundary category label of each pixel in the sample image, and the boundary category label of the pixel includes one of a boundary label or a non-boundary label, the boundary label indicates that the pixel is located in the boundary area on the first side of the text area to be identified, and the non-boundary label indicates that the pixel is not located in the boundary area on the first side of the text area to be identified.

[0108] Correspondingly, when training the initial neural network model, each sample image can be input into the initial neural network model, and the initial neural network model outputs the text category prediction result of each pixel point in each sample image (that is, whether the pixel point is located in the text area to be identified) and the boundary category prediction result (that is, whether the pixel point is located in the boundary area on the first side of the text area to be identified). Then, the value of the second loss function can be obtained based on the text category label and text category prediction result of each pixel point in each sample image, and the value of the first loss function can be obtained based on the boundary category label and boundary category prediction result of each pixel point in each sample image, and the value of the total loss can be obtained based on the value of the first loss function and the value of the second loss function.

[0109] Optionally, the total loss function can be expressed as follows:

[0110] L(p cls ,p side , g cls , g side )=L cls (p cls , g cls )+[g cls>0]L side (p side , g side )

[0111] Among them, L(p cls ,p side ,g cls ,g side ) represents the total loss function, L cls (p cls ,g cls ) represents the second loss function, L side (p side ,g side ) represents the first loss function, p cls Represents the text category prediction result of the pixel point, g cls Indicates the actual text category result corresponding to the text category label of the pixel point (i.e., the annotation result of the region category), p side Represents the boundary category prediction result of the pixel point, g side Indicates the actual boundary category result corresponding to the boundary category label of the pixel point (i.e. the annotation result of the region boundary information), [g cls >0]L side (p side , g side ) indicates that the first loss function is determined based only on the pixels whose text category is text.

[0112] Optionally, the first loss function may adopt K-type cross entropy loss, which can be specifically expressed by the following formula:

[0113]

[0114] Among them, p side Represents the boundary category prediction result of the pixel point, g side Indicates the actual boundary category result corresponding to the boundary category label of the pixel point (i.e., the annotation result of the region boundary information), L side (p side , g side ) represents the first loss function, K represents the total K boundary categories, g k Indicates the actual boundary category result corresponding to the boundary category label of the K-th pixel in the sample image when it is in the K-class boundary category, p k Indicates the boundary category prediction result when the K-th pixel in the sample image is in the K-class boundary category.

[0115] The second loss function can be expressed by the following formula:

[0116]

[0117] Among them, pcls Represents the text category prediction result of the pixel point, g cls Indicates the actual text category result corresponding to the text category label of the pixel point (i.e., the annotation result of the region category), L cls (p cls , g cls ) represents the second loss function, g i represents the true text category result (i.e., the region category annotation result) corresponding to the text category label of the i-th pixel in the sample image, p i Indicates the text category prediction result of the i-th pixel in the sample image. Optional, p i It can be the probability that the i-th pixel in the sample image is a pixel in the text area. At this time, when p i When the value of is greater than the preset value, it can be determined that the text category prediction result of the i-th pixel point is a pixel point in the text area; otherwise, it is considered that the text category prediction result of the i-th pixel point is a pixel point in the background area.

[0118] Accordingly, in an embodiment of the present application, when the text area to be identified in the text image to be identified and the area boundary information of the text area to be identified are obtained based on the area prediction module, the area prediction module can determine the text category label of each pixel point in the text image to be identified based on the input text feature map; accordingly, when the text category label of each pixel point in the text image to be identified is known, it is possible to know which specific pixels are located in the text area to be identified (i.e., determine the text pixels) based on the text category label of each pixel point, and the text area to be identified can be obtained based on the obtained text pixels.

[0119] Optionally, the region prediction module can also determine the boundary category label of each pixel in the text image to be identified based on the text feature map. At this time, based on the boundary category label of each pixel, it can be known which specific pixels are located in the boundary area on the first side of the text area to be identified, and then the boundary information of the first side of the text area to be identified can be obtained based on the pixel composition of the boundary area on the first side of the text area to be identified.

[0120] Among them, in order to reduce the amount of data processing, the region prediction module can only determine the boundary category label of each pixel point located in the text area to be identified when determining the boundary category label of each pixel point in the text image to be identified based on the text feature map, that is, only determine the boundary category label of each text pixel point. Since the number of text pixels is less than the number of all pixels in the text image to be identified, only determining the boundary category label of each text pixel point at this time can effectively reduce the amount of data processing compared to determining the boundary category label of each pixel point in the text image to be identified.

[0121] The method proposed in the embodiment of the present application can quickly annotate curved text samples, which can greatly reduce the annotation cost while ensuring the annotation accuracy.

[0122] Optionally, the method provided in the embodiment of the present application can be used in a curved text detection and recognition system, such as for curved text recognition in an OCR system, thereby enhancing the scene recognition capability of the OCR system. In order to better understand the method provided in the embodiment of the present application, the method is described below in conjunction with a specific application scenario. In this example, the text area in the text image to be recognized and the boundary information of the text area can be obtained based on the regional prediction model, and then text recognition can be performed on the text in the text area to obtain the corresponding text recognition result. Among them, the writing direction of the text in the text image to be recognized is from left to right, and the boundary information of the first side of the text area refers to the left boundary information and the right boundary information of the text area, and the boundary information of the second side of the text area refers to the upper boundary information and the lower boundary information of the text area.

[0123] Among them, the regional prediction model is obtained after training the initial neural network model based on each sample image, and for any sample image, each pixel in the sample image has a corresponding text category label and boundary category label. Among them, the text category label is specifically a background label or a text label, and the boundary category label is specifically a boundary label or a non-boundary label. It can be understood that in this example, the boundary label includes a left boundary label and a right boundary label, which respectively indicate that the pixel point is located at the left boundary of the text area and the right boundary of the text area, and the left boundary label is represented by 1, the right boundary label is represented by 2, the non-boundary label is represented by 0, the background label is represented by 0, and the text label is represented by 1.

[0124] Optional, combine the following Figure 4a to Figure 4d The method of labeling the text category label and boundary category label of each pixel in the sample image is described in detail. Assume that a sample image is as follows Figure 4a As shown, the sample image includes the curved text "XX County YY General Association ZZ Branch" and the rectangular text "AAA Station". The area corresponding to the curved text "XX County YY General Association ZZ Branch" is the curved text area, and the area corresponding to the rectangular text "AAA Station" is the rectangular text area. Furthermore, the upper left corner of each text line can be used as the starting point to annotate each pixel in the sample image with a text category label. Specifically, the text category label of the pixel located inside the polygon (i.e., the rectangular text area and the curved text area) can be set to 1 (i.e., Figure 4b The text category labels of the pixels in the white area of ​​​​the sample image are set to 1), and the text category labels of the pixels outside the polygon in the sample image are set to 0 (that is, Figure 4b The text category labels of the pixels in the black area are set to 0).

[0125] Furthermore, the Douglas-Peuker algorithm can be used to perform curve point thinning on the pixels in the rectangular text area and the curved text area to obtain the left and right boundaries of the rectangular text area and the left and right boundaries of the curved text area (e.g. Figure 4c In this example, we take the labeling of a curved text area as an example. At this time, we can use the determined left boundary (the length of the left boundary is h) as the base of the triangle and shrink an isosceles triangle with a height of h / 2 into the curved text area (as shown in the figure). Figure 4d As shown), an isosceles triangle located inside the curved text area and corresponding to the left boundary of the curved text area is obtained (as shown Figure 4d Similarly, we can get an isosceles triangle located inside the curved text area and corresponding to the right boundary of the curved text area (as shown in C1). Figure 4d ), and an isosceles triangle located inside the rectangular text area and corresponding to the left edge of the rectangular text area (as shown in C2 in FIG. Figure 4d ) and an isosceles triangle located inside the rectangular text area and corresponding to the right edge of the rectangular text area (as shown in C3 in FIG. Figure 4d (as shown in C4 in the figure), then the boundary category labels of the pixels located in the isosceles triangle inside the curved text area and corresponding to the left boundary of the curved text area and the isosceles triangle inside the rectangular text area and corresponding to the left boundary of the rectangular text area are set to the left boundary label 1, the boundary category labels of the pixels located in the isosceles triangle inside the curved text area and corresponding to the right boundary of the curved text area and the isosceles triangle inside the rectangular text area and corresponding to the right boundary of the rectangular text area are set to the right boundary label 2, and the boundary category labels of the pixels in other areas of the sample image except the isosceles triangles are set to the non-boundary label 0.

[0126] Optionally, the network structure of the trained regional prediction model can be as follows Figure 5 As shown, the region prediction model may include a feature extraction module and a region prediction module. The feature extraction module is composed of a CNN+FPN network. The region prediction module includes a position prediction branch and a text end prediction branch. The position prediction branch is used to predict whether the text category label of the pixel point in the text image to be identified is a background label or a text label. The text end prediction branch is used to predict whether the boundary category label of the pixel point in the text image to be identified is a left boundary label, a right boundary label or a non-boundary label.

[0127] Correspondingly, when the text image to be recognized including the curved text area is input into the region prediction model, the CNN network in the feature extraction module performs downsampling feature extraction to obtain a feature map and inputs it into the FPN network. The FPN network performs upsampling feature extraction on the input feature map to obtain a text feature map of the text image to be recognized and inputs it into the region prediction module; the position prediction branch in the region prediction module determines the text category label of each pixel in the text image to be recognized based on the input text feature map. At this time, the text area (such as Figure 5 The text category label of the pixels in the two rectangular areas and the curved area in area A is text label 1, and the text category label of the pixels in other areas (such as Figure 5 The text category label of the pixels in the area A (except the two rectangular areas and the curved area) is the background label 0. The text end prediction branch in the area prediction module determines the boundary category label of each pixel in the text image to be recognized. At this time, the pixel in the left boundary area of ​​the text area to be recognized (such as Figure 5 The boundary category label of each pixel point in the triangle area of ​​​​the B area is the left boundary label 1, and the boundary category label of each pixel point in the right boundary area of ​​the text area to be identified (such as Figure 5 The boundary category label of each pixel point in the triangle area (shown in area C) is the right boundary label 2, while the text category label of the pixel points in other areas is the non-boundary label 0.

[0128] Further, such as Figure 6 As shown, the text category label of each pixel (i.e. Figure 6 text / background in ), through CCL (i.e. Figure 6 CCL_1 in the text image to be recognized) to obtain the curved text area to be recognized (i.e. Figure 6 Then, according to the boundary category label of each pixel point in the curved text area to be identified (i.e. Figure 6 The left / right side of the polygon P is obtained to obtain the region attribution information of all pixels in the polygon P (that is, to determine the pixels belonging to the left boundary of the polygon P and the pixels belonging to the right boundary of the polygon P). According to the pixels belonging to the left boundary of the polygon P, the CCL (that is, Figure 6 CCL_2 in the figure), determine the triangle including the left boundary, and then use the base of the triangle including the left boundary as the left boundary of the polygon P, and according to the pixel points belonging to the right boundary of the polygon P, use CCL (i.e. Figure 6 CCL_2 in the figure), determine the triangle including the right boundary, and then use the bottom edge of the triangle including the right boundary as the right boundary of polygon P. Then, the upper and lower boundaries of polygon P can be determined based on polygon P and the left and right boundaries of polygon P (i.e. Figure 6Determine the upper and lower boundaries of P), and fit the upper and lower boundaries of polygon p with Bezier (i.e. Figure 6 Then, the polygon p is corrected into a rectangle using Bezier cutout and output (i.e. Figure 6 ); further, text recognition can be performed on the text within the output rectangular area to obtain a corresponding text recognition result.

[0129] Optionally, in another example, Figure 7 As shown, assuming that the text image to be recognized includes the curved text "XX County YY General Association ZZ Branch" and the rectangular text "AAA Station", the text in the text image to be recognized can be recognized by the currently commonly used method and the method provided in the embodiment of the present application. Among them, when the currently commonly used method is used to recognize the text in the text image to be recognized, the area where the text is located in the text image to be recognized is directly located, and then text recognition is performed directly. At this time, due to the large amount of background noise in the curved text, it will not be recognized, and only the text recognition result "AAA Station" in the rectangular text can be obtained; and when the method provided in the embodiment of the present application is used to recognize the text in the text image to be recognized, the area of ​​the curved text in the text image to be recognized can be corrected to a rectangular area, and then text recognition is performed. At this time, not only the text recognition result "AAA Station" in the rectangular text can be obtained, but also the text recognition result "XX County YY General Association ZZ Branch" in the curved text can be obtained.

[0130] It can be seen that the method provided in the embodiment of the present application can correct the output form of the detection results of curved text in the image to be recognized into a rectangular form, thereby reducing background noise, improving the text recognition effect, and improving the robustness of the OCR system. It can also be flexibly added to the existing text detection algorithm, such as adding the above-mentioned region prediction module at the end of the detection network, and can be trained in an end-to-end manner.

[0131] The present application embodiment provides a text recognition device, such as Figure 8 As shown, the text recognition device 60 may include: an image acquisition module 601, a feature extraction module 602, a boundary information determination module 603, a region correction module 604 and a text recognition module 605, wherein:

[0132] An image acquisition module 601 is used to acquire a text image to be recognized, wherein the image to be processed includes a curved text area;

[0133] Feature extraction module 602, used to extract text feature map of the text image to be recognized;

[0134] A boundary information determination module 603 is configured to determine a text region to be identified in an image and region boundary information of the text region to be identified based on the text feature map, wherein the text region to be identified includes a curved text region;

[0135] A region correction module 604 is used to correct the text region to be recognized according to the region boundary information to obtain a corrected rectangular text region;

[0136] The text recognition module 605 is used to perform text recognition on the rectangular text area to obtain a text recognition result of the text image to be recognized.

[0137] Optionally, the region boundary information includes boundary information of a first side of the text region to be recognized, wherein the first side is two sides consistent with the writing direction of the text in the text image to be recognized;

[0138] When correcting the text area to be recognized based on the area boundary information, the area correction module is specifically used to:

[0139] Determining boundary information of a second side of the text area to be recognized based on the boundary information of the first side and the text area to be recognized, wherein the second side is two sides in a direction perpendicular to the writing direction;

[0140] The text area to be recognized is corrected according to the boundary information of the first side and the boundary information of the second side.

[0141] Optionally, when determining the text region to be recognized and the region boundary information of the text region to be recognized in the text image to be recognized based on the text feature map, the boundary information determination module is specifically configured to:

[0142] Based on the text feature map, determine the text pixels in the text image to be recognized and the boundary pixels of the text pixels belonging to the text area to be recognized;

[0143] Determine the text area to be recognized in the text image to be recognized based on the text pixels;

[0144] Based on the sum of boundary pixels, the region boundary information of the text image to be recognized is determined.

[0145] Optionally, the region boundary information includes boundary information of a first side of the text region to be recognized, the first side being two sides consistent with the writing direction of the text in the text image to be recognized, the boundary pixel points including boundary pixel points on one side of the first side and boundary pixel points on the other side of the first side, and the boundary information determination module, when determining the region boundary information of the text image to be recognized based on the boundary pixel points, is specifically configured to:

[0146] Determine a first boundary connected domain on one side according to boundary pixel points on one side of the first side, and determine boundary information on one side of the first side based on the first boundary connected domain;

[0147] A second boundary connected domain on the other side is determined according to boundary pixel points on the other side of the first side, and boundary information of the other side of the first side is determined based on the second boundary connected domain.

[0148] Optionally, when the boundary information determining module determines the boundary information of the second side of the text region to be recognized based on the boundary information of the first side of the text region to be recognized and the text region to be recognized, it is specifically configured to:

[0149] Determining boundary key points of a second side of the text area to be recognized based on the boundary information of the first side and the text area to be recognized;

[0150] Boundary information of a second side of the to-be-recognized text region is determined based on the boundary key points and the boundary information of the first side.

[0151] Optionally, when the boundary information determining module determines the boundary information of the second side of the text area to be recognized based on the boundary key points and the boundary information of the first side, it is specifically configured to:

[0152] Determining initial boundary information of a second side of the text area to be recognized based on the boundary key points and the boundary information of the first side;

[0153] The initial boundary information is subjected to curve smoothing processing to obtain boundary information of the second side of the text area to be recognized.

[0154] Optionally, when determining the text region to be recognized and the region boundary information of the text region to be recognized in the text image to be recognized based on the text feature map, the boundary information determination module is specifically configured to:

[0155] Based on the text image to be recognized, a text feature map is extracted through the feature extraction module of the region prediction model;

[0156] Based on the text feature map, the region prediction module of the region prediction model is used to obtain the text region to be recognized in the text image to be recognized and the region boundary information of the text region to be recognized;

[0157] Among them, the regional prediction model is obtained through the following methods:

[0158] Obtain a sample dataset, the sample dataset including each sample image, and a text category label and a boundary category label of each sample image, the text category label including a labeling result of a region category in the sample image, the boundary category label including a labeling result of region boundary information of a text region in the sample image, the text region in the sample image including a curved text region, and the region category including a text region and a background region;

[0159] The initial neural network model is trained based on the sample data set until the total loss function corresponding to the initial neural network model converges, thereby obtaining a regional prediction model;

[0160] Among them, the total loss function includes a first loss function and a second loss function. The value of the first loss function represents the difference between the annotation results of the region boundary information of the text area in each sample image and the predicted region boundary information output by the model. The second loss function represents the difference between the annotation results of the region category in each sample image and the predicted results of the region category output by the model.

[0161] Optionally, for any sample image, the text category label includes a text category label of each pixel in the sample image, where the text category label of the pixel is a background label or a text label; the boundary category label includes a boundary category label of each pixel in the sample image, where the boundary category label of the pixel is a boundary label or a non-boundary label;

[0162] The input of the initial neural network model is each sample image, and the output of the initial neural network model is the text category prediction result and boundary category prediction result of each pixel point in each sample image;

[0163] The value of the first loss function represents the difference between the boundary category label and the boundary category prediction result of each pixel point in each sample image, and the value of the second loss function represents the difference between the text category label and the text category prediction result of each pixel point in each sample image.

[0164] The text recognition device of the embodiment of the present application can execute a text recognition method provided by the embodiment of the present application. The implementation principle is similar and will not be repeated here.

[0165] The text recognition device of the embodiment of the present application can be a computer program (including program code) running on a computer device, for example, the text recognition device is an application software; the device can be used to execute the corresponding steps in the text recognition method provided in the embodiment of the present application.

[0166] In some embodiments, the text recognition device provided in the embodiments of the present application can be implemented by a combination of software and hardware. As an example, the text recognition device provided in the embodiments of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the text recognition method provided in the embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0167] In other embodiments, the text recognition device provided in the embodiments of the present application can be implemented in software. Figure 8 A text recognition device 60 is shown, which can be software in the form of a program and plug-in, and includes a series of modules, including an image acquisition module 601, a feature extraction module 602, a boundary information determination module 603, a region correction module 604 and a text recognition module 605; wherein the image acquisition module 601, the feature extraction module 602, the boundary information determination module 603, the region correction module 604 and the text recognition module 605 are used to implement the text recognition method provided in the embodiment of the present application.

[0168] The present application embodiment provides an electronic device, such as Figure 9 As shown, Figure 9 The electronic device 2000 shown includes a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, for example, via a bus 2002. Optionally, the electronic device 2000 may further include a transceiver 2004. It should be noted that in actual applications, the number of transceivers 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0169] The processor 2001 is used in the embodiment of the present application to implement Figure 8 The functions of each module are shown.

[0170] Processor 2001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0171] The bus 2002 may include a path for transmitting information between the above components. The bus 2002 may be a PCI bus or an EISA bus, etc. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0172] The memory 2003 may be a ROM or other type of static storage device that can store static information and computer programs, a RAM or other type of dynamic storage device that can store information and computer programs, or an EEPROM, a CD-ROM or other optical disk storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store or in the form of a data structure the desired computer program and can be accessed by a computer, but is not limited thereto.

[0173] The memory 2003 is used to store the computer program for executing the application program of the present application solution, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the computer program of the application program stored in the memory 2003 to implement the actions of the text recognition device provided in the embodiment shown in Figure 8.

[0174] An embodiment of the present application provides an electronic device, including a processor and a memory: the memory is configured to store a computer program, and when the computer program is executed by the processor, the processor performs any one of the methods in the above embodiments.

[0175] An embodiment of the present application provides a computer-readable storage medium for storing a computer program. When the computer program is run on a computer, the computer can execute any one of the methods in the above embodiments.

[0176] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text recognition method provided in the various optional implementations described above.

[0177] The nouns and implementation principles involved in a computer-readable storage medium in this application can be specifically referred to a text recognition method in an embodiment of this application, and will not be repeated here.

[0178] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0179] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A text recognition method, characterized in that: include: Acquire a text image to be recognized, wherein the text image to be recognized includes a curved text area; Extracting a text feature map of the text image to be recognized; Based on the text feature map, determining text pixels in the to-be-recognized text image and boundary pixels of the text pixels belonging to a to-be-recognized text region, wherein the to-be-recognized text region includes the curved text region; Determining a text area to be recognized in the text image to be recognized based on the text pixels; Determining the region boundary information of the text image to be recognized based on the boundary pixel points; Correcting the text area to be recognized according to the area boundary information to obtain a corrected rectangular text area; Performing text recognition on the rectangular text area to obtain a text recognition result of the text image to be recognized; The region boundary information includes boundary information of a first side of the to-be-recognized text region, the first side being two sides consistent with a writing direction of the text in the to-be-recognized text image, and the boundary pixel points include boundary pixel points on one side of the first side and boundary pixel points on the other side of the first side; The region boundary information is determined by the following steps: Determine, based on an area where boundary pixels of one side of the first side are located, a first boundary connected region in which the side is a triangle, and use a base of the first boundary connected region as boundary information of the one side of the first side; According to the area where the boundary pixels of the other side of the first side are located, a second boundary connected region of the other side is determined to be a triangle, and a bottom of the second boundary connected region is used as boundary information of the other side of the first side.

2. The method according to claim 1, characterized in that Correcting the to-be-recognized text area according to the area boundary information includes: determining boundary information of a second side of the text region to be recognized based on the boundary information of the first side of the text region to be recognized and the text region to be recognized, wherein the second side is two sides in a direction perpendicular to the writing direction; The to-be-recognized text area is corrected according to the boundary information of the first side and the boundary information of the second side.

3. The method according to claim 2, characterized in that The determining, based on the boundary information of the first side of the text region to be recognized and the text region to be recognized, the boundary information of the second side of the text region to be recognized includes: Determining boundary key points of a second side of the text area to be recognized based on the boundary information of the first side and the text area to be recognized; Boundary information of a second side of the to-be-recognized text area is determined based on the boundary key points and the boundary information of the first side.

4. The method according to claim 3, characterized in that The determining, based on the boundary key points and the boundary information of the first side, boundary information of the second side of the to-be-recognized text area includes: determining initial boundary information of a second side of the to-be-recognized text area according to the boundary key points and the boundary information of the first side; Perform curve smoothing processing on the initial boundary information to obtain boundary information of the second side of the text area to be recognized.

5. The method according to any one of claims 1 to 4, characterized in that The text feature map is extracted based on the text image to be recognized by a feature extraction module of a region prediction model; The region boundary information is obtained based on the text feature map through the region prediction module of the region prediction model; The regional prediction model is obtained by: Acquire a sample dataset, the sample dataset including each sample image, and a text category label and a boundary category label for each sample image, the text category label including a labeling result of a region category in the sample image, the boundary category label including a labeling result of region boundary information of a text region in the sample image, the text region in the sample image including a curved text region, and the region category including a text region and a background region; Training the initial neural network model based on the sample data set until the total loss function corresponding to the initial neural network model converges, thereby obtaining the regional prediction model; Among them, the total loss function includes a first loss function and a second loss function. The value of the first loss function represents the difference between the annotation results of the region boundary information of the text area in each sample image and the predicted region boundary information output by the model. The second loss function represents the difference between the annotation results of the region category in each sample image and the predicted results of the region category output by the model.

6. The method according to claim 5, characterized in that For any of the sample images, the text category label includes a text category label for each pixel in the sample image, where the text category label for each pixel is a background label or a text label; the boundary category label includes a boundary category label for each pixel in the sample image, where the boundary category label for each pixel is a boundary label or a non-boundary label; The input of the initial neural network model is each of the sample images, and the output of the initial neural network model is the text category prediction result and the boundary category prediction result of each pixel point in each of the sample images; The value of the first loss function represents the difference between the boundary category label and the boundary category prediction result of each pixel point in each of the sample images, and the value of the second loss function represents the difference between the text category label and the text category prediction result of each pixel point in each of the sample images.

7. The method according to claim 6, characterized in that The boundary label includes the boundary label of the first side; for each sample image, the pixel point whose boundary category label is the boundary label of the first side is determined by the following method: A triangular region inside the text region is constructed with the boundary of the first side in the sample image as the base, and each pixel in the triangular region is used as a pixel point whose boundary category label is the boundary label of the first side.

8. A text recognition device, characterized in that: include: An image acquisition module, configured to acquire an image of text to be recognized, wherein the image of text to be recognized includes a curved text area; A feature extraction module, configured to extract a text feature map of the text image to be recognized; a boundary information determination module, configured to determine, based on the text feature map, text pixels in the to-be-recognized text image and boundary pixels belonging to the to-be-recognized text region among the text pixels; determine, based on the text pixels, the to-be-recognized text region in the to-be-recognized text image; and determine, based on the boundary pixels, region boundary information of the to-be-recognized text image; the to-be-recognized text region includes the curved text region; wherein the region boundary information includes boundary information of a first side of the to-be-recognized text region, the first side being two sides consistent with a writing direction of the text in the to-be-recognized text image, and the boundary pixels including boundary pixels on one side of the first side and boundary pixels on the other side of the first side; The region boundary information is determined by the following steps: Determine, based on an area where boundary pixels on one side of the first side are located, a first boundary connected region that is a triangle on the one side, and use the base of the first boundary connected region as boundary information for the one side of the first side; determine, based on an area where boundary pixels on the other side of the first side are located, a second boundary connected region that is a triangle on the other side, and use the base of the second boundary connected region as boundary information for the other side of the first side; A region correction module, configured to correct the text region to be recognized according to the region boundary information to obtain a corrected rectangular text region; The text recognition module is used to perform text recognition on the rectangular text area to obtain a text recognition result of the text image to be recognized.

9. The device according to claim 8, characterized in that The region correction module is specifically configured to: determining boundary information of a second side of the text region to be recognized based on the boundary information of the first side of the text region to be recognized and the text region to be recognized, wherein the second side is two sides in a direction perpendicular to the writing direction; The to-be-recognized text area is corrected according to the boundary information of the first side and the boundary information of the second side.

10. The device according to claim 9, characterized in that The boundary information determining module is specifically configured to, when determining the boundary information of the second side of the text region to be recognized based on the boundary information of the first side of the text region to be recognized and the text region to be recognized: Determining boundary key points of a second side of the text area to be recognized based on the boundary information of the first side and the text area to be recognized; Boundary information of a second side of the to-be-recognized text area is determined based on the boundary key points and the boundary information of the first side.

11. The device according to claim 10, characterized in that When the boundary information determining module determines the boundary information of the second side of the to-be-recognized text area based on the boundary key points and the boundary information of the first side, the boundary information determining module is specifically configured to: determining initial boundary information of a second side of the to-be-recognized text area according to the boundary key points and the boundary information of the first side; Perform curve smoothing processing on the initial boundary information to obtain boundary information of the second side of the text area to be recognized.

12. The device according to any one of claims 8 to 11, characterized in that The text feature map is extracted based on the to-be-recognized text image by a feature extraction module of a region prediction model; the region boundary information is obtained based on the text feature map by a region prediction module of the region prediction model; The regional prediction model is obtained by: Acquire a sample dataset, the sample dataset including each sample image, and a text category label and a boundary category label for each sample image, the text category label including a labeling result of a region category in the sample image, the boundary category label including a labeling result of region boundary information of a text region in the sample image, the text region in the sample image including a curved text region, and the region category including a text region and a background region; Training the initial neural network model based on the sample data set until the total loss function corresponding to the initial neural network model converges, thereby obtaining the regional prediction model; Among them, the total loss function includes a first loss function and a second loss function. The value of the first loss function represents the difference between the annotation results of the region boundary information of the text area in each sample image and the predicted region boundary information output by the model. The second loss function represents the difference between the annotation results of the region category in each sample image and the predicted results of the region category output by the model.

13. The device according to claim 12, characterized in that For any of the sample images, the text category label includes a text category label for each pixel in the sample image, where the text category label for each pixel is a background label or a text label; the boundary category label includes a boundary category label for each pixel in the sample image, where the boundary category label for each pixel is a boundary label or a non-boundary label; The input of the initial neural network model is each of the sample images, and the output of the initial neural network model is the text category prediction result and the boundary category prediction result of each pixel point in each of the sample images; The value of the first loss function represents the difference between the boundary category label and the boundary category prediction result of each pixel point in each of the sample images, and the value of the second loss function represents the difference between the text category label and the text category prediction result of each pixel point in each of the sample images.

14. The device according to claim 13, characterized in that The boundary label includes the boundary label of the first side; for each sample image, the pixel point whose boundary category label is the boundary label of the first side is determined by the following method: A triangular region inside the text region is constructed with the boundary of the first side in the sample image as the base, and each pixel in the triangular region is used as a pixel point whose boundary category label is the boundary label of the first side.

15. An electronic device, characterized in that: Including processor and memory: The memory is configured to store a computer program, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program. When the computer program is run on a computer, the computer can execute the method according to any one of claims 1 to 7.

Images