Text recognition method, device, equipment, readable storage medium and program product

Abstract

The embodiments of the present application propose a text recognition method, apparatus, device, readable storage medium, and program product, which can be applied to fields or scenarios such as cloud technology, artificial intelligence, smart platforms, application software, in-vehicle, and text recognition. The method includes: obtaining an original image; segmenting the original image to obtain at least one first text area image; calling a text recognition model to perform text recognition on the at least one first text area image to obtain the target text in the original image. The text recognition model is obtained by training the first neural network and the second neural network included in the initial recognition model based on the sample image and first supervisory information. The first supervisory information includes a character encoding reference sequence obtained by processing the reference text corresponding to the sample image based on character encoding rules. Through the embodiments of the present application, the accuracy of agglutinative text recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text recognition method, a text recognition apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous development of the internet, the amount of content in minority languages ​​is increasing. How to make this content available for search engines, public opinion data analysis, and other applications has attracted significant attention. Currently, there are complex agglutinative languages ​​whose letter forms change depending on their position within a word. When using non-agglutinative text recognition methods for text recognition, data sparsity and other issues can easily occur, resulting in low recognition accuracy for agglutinative text. Therefore, accurately identifying agglutinative text in images is a pressing issue. Summary of the Invention

[0003] The present application provides a text recognition method, apparatus, device, readable storage medium, and program product, which can improve the accuracy of agglutinative text recognition.

[0004] In a first aspect, the present application provides a text recognition method, the method comprising:

[0005] Get the original image;

[0006] Segmenting the original image to obtain at least one first text region image;

[0007] A text recognition model is called to perform text recognition on the at least one first text area image to obtain the target text in the original image. The text recognition model is obtained by training the first neural network and the second neural network included in the initial recognition model based on the sample image and the first supervisory information. The first supervisory information includes a character encoding reference sequence obtained by processing the reference text corresponding to the sample image based on the character encoding rules.

[0008] In a second aspect, the present application provides a text recognition device, the device comprising:

[0009] An acquisition module, used to acquire the original image;

[0010] a processing module, configured to segment the original image to obtain at least one first text region image;

[0011] The recognition module is used to call a text recognition model to perform text recognition on the at least one first text area image to obtain the target text in the original image. The text recognition model is obtained by training the first neural network and the second neural network included in the initial recognition model based on the sample image and the first supervision information. The first supervision information includes a character encoding reference sequence obtained by processing the reference text corresponding to the sample image based on the character encoding rules.

[0012] In a third aspect, the present application provides a computer device comprising: a processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are interconnected, wherein the storage device stores executable program code, and the processor is used to call the executable program code to implement the above-mentioned text recognition method.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions are executed by a processor to implement the text recognition method as described above.

[0014] In a fifth aspect, the present application provides a computer program product, which includes a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the above-mentioned text recognition method.

[0015] The present application obtains an original image containing text content and performs segmentation processing to obtain at least one first text region image. The single first text region image is then processed by a text recognition model to obtain the target text in the original image. This can avoid the mutual influence between multiple lines of text and improve the accuracy of image processing. The text recognition model in the present application is obtained by training an initial recognition model including a first neural network and a second neural network. By processing the first text region image using two neural networks with different network structures, the feature information of the image can be fully utilized, the processing capability of the model can be improved, and thus the accuracy of text recognition can be improved. During the model training process, the reference text corresponding to the sample image is mapped to a character encoding reference sequence using the established character encoding rules, which serves as supervision information for model training. The above character encoding rules can flexibly map and transform the text corresponding to the target language according to the grammatical features of the target language, so that the converted character encoding reference sequence can flexibly distinguish and represent characters with different deformation forms. The above method can be well applied to the text recognition processing of languages ​​with deformation forms, such as agglutinative languages. Integrating character encoding rules into the text recognition process and using the text recognition model to process the character encoding reference sequence corresponding to the text can improve the accuracy of the recognition results of agglutinative language text. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1A is a schematic diagram of a text recognition process provided by an exemplary embodiment of the present application;

[0018] Figure 1B is a schematic diagram of another text recognition process provided by an exemplary embodiment of the present application;

[0019] Figure 1C is a schematic diagram of another text recognition process provided by an exemplary embodiment of the present application;

[0020] Figure 1D This is a schematic diagram of the architecture of a text recognition system provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of a text recognition method provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a flowchart of another text recognition method provided by an exemplary embodiment of the present application;

[0023] Figure 4A is a flowchart of a training text recognition model provided by an exemplary embodiment of the present application;

[0024] Figure 4B is a schematic diagram of a selection rule framework provided by an exemplary embodiment of the present application;

[0025] Figure 4C is a flowchart of an application text recognition model provided by an exemplary embodiment of the present application;

[0026] Figure 5 is a schematic block diagram of a text recognition device provided by an exemplary embodiment of the present application;

[0027] Figure 6 It is a schematic block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] It should be noted that the terms "first" and "second" in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0030] The embodiments of the present invention can be applied to various fields or scenarios such as cloud computing, cloud Internet of Things, cloud gaming, artificial intelligence, in-vehicle scenarios, smart transportation, and assisted driving. Several typical fields or scenarios will be introduced below.

[0031] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, machine learning / deep learning, etc. The solution provided in the embodiment of the present application involves technologies such as machine learning and computer vision technology under artificial intelligence technology, which will be described below.

[0032] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. The present application mainly relates to artificial neural networks in machine learning technology. Specifically, the method proposed in the present application obtains the image features of the sample text image through the artificial neural network in the text detection model during the training process, and then determines the loss based on the annotation information of the sample image, and optimizes the text detection model through the loss.

[0033] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, virtual reality, augmented reality, simultaneous positioning and map construction, autonomous driving, smart transportation and other technologies. Specifically, the method proposed in this application uses image recognition technology to perform image segmentation, text recognition and other processing on the original text image, thereby obtaining the target text in the original text image.

[0034] With the advancement of AI research and technology, AI is being studied and applied in a variety of fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, robots, smart healthcare, intelligent customer service, connected vehicles, autonomous driving, and 3D gaming. As technology develops, AI will be applied in even more areas and play an increasingly important role.

[0035] The text detection and text recognition model constructed in this application includes a continuous temporal classification network and a bidirectional long short-term memory network. In order to better understand the solution of the embodiment of this application, the relevant technologies involved in the embodiment of this application are first introduced below.

[0036] Long Short-Term Memory (LSTM) is a time-recurrent neural network specifically designed to address the long-term dependency issues of typical recurrent neural networks (RNNs). All RNNs have a chain-like structure of repeating neural network modules. In a standard RNN, this repeating module has a very simple structure, such as a tanh layer. (LSTM is primarily designed to address the problems of vanishing and exploding gradients during long sequence training. Simply put, compared to ordinary RNNs, LSTMs can perform better on longer sequences.) LSTM networks have two variants: bidirectional recurrent neural networks and deep recurrent neural networks. The main structure of a bidirectional recurrent neural network is composed of two unidirectional recurrent neural networks.

[0037] CTC (Connectionist Temporal Classification) is a neural network-based time series classification tool that can be used for sequence modeling. Training samples do not require alignment, and its core is the definition of a special objective function. For example, supervised learning is required in the training of speech recognition acoustic models. The introduction of CTC relaxes this one-to-one correspondence requirement, requiring only one input sequence and one output sequence for training. CTC has the following advantages: it does not require data alignment and one-to-one labeling; and it directly outputs the probability of sequence prediction, requiring no external post-processing. CTC solves this problem by adding a blank symbol to the set of labeled symbols, then using an RNN for labeling, and finally eliminating the blank symbol and the predicted duplicate symbols.

[0038] The attention mechanism in neural networks is a resource allocation solution that, when computing power is limited, allocates computing resources to more important tasks while also addressing the problem of information overload. In neural network learning, generally speaking, the more parameters a model has, the stronger its expressive power and the greater the amount of information it can store. However, this can lead to information overload. By introducing an attention mechanism, the attention mechanism focuses on the most critical information for the current task among the numerous inputs, reducing attention to other information and even filtering out irrelevant information. This can address information overload and improve the efficiency and accuracy of task processing. This mechanism allows limited attention resources to be used to quickly filter out high-value information from a large amount of information.

[0039] Among the languages ​​currently in use, there is a relatively special type of language (referred to as the target language in subsequent embodiments). The target language is an agglutinative language. An agglutinative language refers to a language that mainly relies on affixes to express the grammatical meaning of words, for example, through prefixes, infixes, suffixes and other affixes to express the grammatical meaning of words. The main feature of agglutinative languages ​​is that there is no internal inflection. Each morpheme only represents one grammatical meaning, and each grammatical meaning is always represented by a morpheme. Therefore, if a word wants to represent three grammatical meanings, it needs to have three morphemes. In addition, the connection between the root and the morpheme of an agglutinative language is not tight, and both have considerable independence. Therefore, when using non-agglutinative language text recognition methods to perform text recognition on the target language, problems such as data sparsity are prone to occur, resulting in low recognition accuracy of agglutinative language text.

[0040] Specifically, the target language has a total of 33 letters, which are divided into two categories: vowels and consonants. The 9 vowels are divided into two categories: 5 front vowels and 4 back vowels. The 24 consonants are divided into two categories: 10 voiceless consonants and 14 voiced consonants. The target language uses two phonetic scripts based on two alphabets. The target language has a total of 105 writing forms. In terms of form, the target language has many letters, many symbols, and complex letter shapes. Each letter is different at the beginning, middle, and end of a word, and is written in four different ways: single upright, back-linked, double-linked, and front-linked. The target language letters are written horizontally from right to left, and the letters are composed of symbols such as vertical strokes, teeth, dots, and hooks.

[0041] With the continuous development of the Internet, the amount of minority language content (such as agglutinative languages) on the Internet is increasing. How to make minority language content on the Internet provide services for applications such as search engines and public opinion data analysis has attracted great attention from the society. The target language is a typical agglutinative language. The letter morphology of the target language is complex and will change into different letter morphologies according to the position in the word. Therefore, how to accurately identify agglutinative text such as the target language in an image is a problem that needs to be solved urgently. The present application first provides a recognition method based on a single character and a recognition method based on a text sequence.

[0042] Recognition based on a single character is mainly divided into two parts: character segmentation and character recognition. Figure 1A Each character is considered to be composed of multiple strokes (for example, the character "HOUSE" in the image). The stroke features are learned through clustering, and then a random forest classifier is used to classify the characters. The limitation of single-character recognition algorithms is that the complex background of natural scene images and the irregular arrangement of text make it very difficult to separate characters from the background, which is the main factor limiting the performance of the entire recognition system.

[0043] When recognizing text sequences, we can treat the text lines as a whole and utilize contextual semantic information for recognition, which can improve recognition accuracy. With the continuous advancement of machine learning technology, a variety of methods and techniques are being developed to process text sequences.

[0044] For example, there are Connectionist Temporal Classification (CTC), Recurrent Neural Network (RNN), and Attention Mechanism. CTC was initially used for automatic alignment of phoneme data in speech recognition, but has since been widely used for character data alignment in image and text recognition, serving as a loss function to optimize network parameters.

[0045] This application proposes to use the convolutional recursive neural network model (RCNN) for text recognition. Figure 1B , the figure consists of a deep convolutional neural network (corresponding to the convolutional layer in the figure) and a recurrent neural network (corresponding to the recurrent layer in the figure). The model structure includes three parts: convolutional layer, recurrent layer and transcription layer. First, a text image is input, and the convolutional neural network is used to extract the deep features of the image (that is, the convolutional feature map in the figure). Then, the recurrent neural network is used to learn the feature sequence of the text semantics, and the prediction result of each frame of the image is obtained through a deep bidirectional long short-term memory network. Finally, the prediction sequence (for example, "statte") is output. In the above process, the continuous temporal classification (CTC) loss function can be used simultaneously to optimize the network parameters. The model can process text sequences of arbitrary length and is suitable for text recognition tasks without a dictionary. It is an end-to-end trainable model.

[0046] This application also proposes a recursive recurrent neural network based on the attention mechanism (Recursive RecurrentNets with Attention Modeling, R2AM). Figure 1C The diagram includes image feature extraction and language processing modules. This model uses a recursive convolutional neural network, which can more efficiently extract image features. Furthermore, the attention mechanism is used for sequence modeling, selectively utilizing image features, and can be trained end-to-end using a standard backpropagation algorithm. This network model can also recognize natural scene text without dictionary constraints.

[0047] The above-mentioned methods are effective for text recognition in non-agglutinative language scenarios (such as English, which has a small number of character categories and lacks agglutination). However, they still present some challenges when applied to agglutinative, minority languages. For example, compared to Chinese, the target language has relatively scarce resources. Furthermore, the target language differs significantly from Chinese in terms of its naming conventions. For example, the target language is a typical agglutinative language with complex morphological variations, which can easily lead to data sparsity. Furthermore, there are currently relatively few scholars researching text recognition in the target language, resulting in significant lags in recognition methods and effectiveness compared to Chinese. The amount of content in the target language on the internet is constantly increasing, and how to make this content available for applications such as search engines and public opinion data analysis has attracted significant attention. To this end, the first challenge is to collect target language text data online. The core of this problem is accurately identifying target language text online.

[0048] The above method, when applied to text recognition in the target language, has the following disadvantages: First, the target language is an agglutinative language with 33 basic letters (semantic characters), each of which has 2-4 variants (inflected characters). When letters in the target language form words, different inflected forms are selected depending on their position within the word, placing high demands on the encoding and decoding rules of the target language. Second, while the decoding framework based on the attention mechanism outperforms the CTC-based decoding method in terms of recognition accuracy, the serial decoding structure of the attention mechanism significantly limits recognition speed and consumes more resources.

[0049] This application proposes a new target language encoding rule mapping method based on the rule characteristics of the target language, and then starts from the character code obtained by mapping, adopts a multi-task collaborative recognition method, which significantly improves the recognition accuracy of the target language while ensuring recognition efficiency. First, the line text box of the target language is detected based on the segmentation text detection method, and then the selection rule framework of the target language is constructed. Then, the encoding and decoding transformation of the target language is performed according to the formulated selection rules, and finally a multi-task collaborative recognition framework is used to identify the text content in each line text box of the target language. The above method significantly improves the recognition accuracy of the target language while ensuring recognition efficiency.

[0050] It can be understood that in the specific implementation of this application, related data such as original images and sample images are involved. When the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0051] This application will be specifically described through the following examples:

[0052] See also Figure 1D , which is a schematic diagram of the architecture of a text recognition system provided by an exemplary embodiment of the present application. The text recognition system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network, a wide area network, a mobile Internet, etc. The operating object operates on the browser or client application of the terminal device 101 to process various image data. The server 102 can provide various text detection, text recognition and other services to the operating object in response to the operation.

[0053] Specifically, the terminal device 101 sends the original image to the server 102; the server 102 can use the text detection model to segment the original image to obtain at least one first text area image, each first text area image corresponds to a text line; the server 102 then calls the text recognition model to perform text recognition on the at least one first text area image to obtain the target text in the original text image; the server 102 finally sends the target text to the terminal device 101 so that the results can be displayed on the terminal device 101.

[0054] In the model training stage, the text detection model is first trained. The server 102 can obtain a sample image from the terminal device 101, and then use the initial detection model to process the text area image corresponding to the sample image to obtain predicted feature data. The parameters of the initial detection model are then adjusted using the second supervision information and the predicted feature data to obtain a text detection model.

[0055] Then the text recognition model is trained. The server processes the text area image corresponding to the sample image by calling the initial recognition model to obtain the character encoding prediction sequence. Then, the parameters of the initial recognition model are adjusted according to the character encoding reference sequence and character encoding prediction sequence of the reference text to obtain the text recognition model.

[0056] The terminal device 101 is also referred to as a terminal, user equipment (UE), access terminal, subscriber unit, mobile device, user terminal, wireless communication device, user agent, or user apparatus. The terminal device may be, but is not limited to, a smart home appliance, a handheld device with wireless communication capabilities (e.g., a smartphone or tablet), a computing device (e.g., a personal computer (PC)), an in-vehicle terminal, an intelligent voice interaction device, a wearable device, or other smart device.

[0057] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0058] It is understandable that the architecture diagram of the system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. For example, the access method provided in the embodiment of the present application can be executed by the server 102, and can also be executed by other servers or server clusters that are different from the server 102 and can communicate with the terminal device 101 and / or the server 102. For example, in the present application, the terminal device 101 and the server 102 can be two independent devices, or they can be the same device (that is, the architecture of the entire text recognition system only includes the terminal device 101, and the terminal device 101 has the ability to operate independently, and independently realizes the task of text recognition by embedding a text detection model and a text recognition model), that is, the terminal device 101 directly obtains the original text image; then the original text image is segmented by the text detection model to obtain at least one first text area image; then the text recognition model on the terminal device 101 is called to perform text recognition on at least one first text area image to obtain the target text in the original text image.

[0059] Those skilled in the art will know that Figure 1D The number of terminal devices and servers in the embodiment is merely illustrative. Any number of terminal devices and servers can be configured according to business implementation needs. Moreover, as the system architecture evolves and new business scenarios emerge, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] In the subsequent embodiments, the terminal device 101 will be referred to as a terminal device, and the server 102 will be referred to as a server, which will not be described in detail in the subsequent embodiments.

[0061] See also Figure 2 , Figure 2 This is a flowchart of a text recognition method provided by an exemplary embodiment of the present application, in which the method is applied to a server (referring to Figure 1D Taking the server 102 in FIG. 1 as an example, the method may include the following steps:

[0062] S201: Acquire an original image.

[0063] In the embodiment of the present application, the original image is the image to be processed for text recognition, and the original image includes text content. The text content in the original image can be text corresponding to a non-adhesive language (such as Chinese, English, etc.) or text corresponding to an agglutinative language (such as the target language, Japanese, etc.).

[0064] In one embodiment, the original image of the present application can be image data obtained by the server through an image acquisition device (such as a camera) on a terminal device, or it can be image data obtained by the server from other devices in the network through the Internet, or it can be image data obtained directly from a local storage device (such as a USB flash drive).

[0065] In one embodiment, the original image may be an image including one line of text data, or an image including multiple lines of text data, or an image including multiple paragraphs of text or divided into multiple text areas.

[0066] S202: Segment the original image to obtain at least one first text region image.

[0067] In this embodiment of the present application, the server segments the original image into at least one first text region image based on the concept of text line segmentation. Each of the at least one first text region image can correspond to a region image corresponding to a line of text in the original image, which is segmented by text line. By obtaining the first text region image of the original image for text recognition processing, the server can avoid mutual influence between multiple lines of text, thereby improving the accuracy of text recognition.

[0068] In one embodiment, the text content in the original image may be deformed (e.g., skewed text, text overlap, etc.) due to external factors during the acquisition process (e.g., when the image acquisition device on the terminal device is capturing the image, the captured object is not placed flat, etc.). Therefore, the server can use the image detection model to perform image correction on the original image (e.g., correcting the text rotation angle, correcting the text overlap area), and then determine the first text area image based on the image-corrected original image, further improving the accuracy of text recognition.

[0069] S203. Call a text recognition model to perform text recognition on the at least one first text area image to obtain the target text in the original image, wherein the text recognition model is obtained by training a first neural network and a second neural network included in an initial recognition model based on a sample image and first supervisory information, and the first supervisory information includes a character encoding reference sequence obtained by processing a reference text corresponding to the sample image based on a character encoding rule.

[0070] In an embodiment of the present application, the server can perform text recognition on each of the at least one first text area image by calling the text recognition model to obtain the target text in the original image. The text recognition model is trained using sample images (including corresponding reference text, that is, annotated text). During the training process of the text recognition model, the server processes the sample images using the initial recognition model to obtain predicted text, and then adjusts the network parameters of the initial recognition model based on the predicted text and the first supervisory information to obtain the text recognition model.

[0071] The first supervisory information includes a character encoding reference sequence, which is obtained by character mapping the reference text corresponding to the sample graphic using character encoding rules. Through the above method, the character encoding reference sequence is used as supervisory information for the text recognition model and combined with the predicted text to complete the training of the text recognition model. Through the above method, when a character has multiple deformation forms, the character can be mapped to different deformation mapping codes according to the character mapping rules to distinguish the character deformation forms in different situations.

[0072] In one embodiment, each sample text image may include a text image, annotated text, and other supervisory information (i.e., annotation information, such as reference pixel type, reference connection type between pixels, and reference rotation angle, etc.). For example, the server may process the first text region image using an initial recognition model to obtain predicted feature data (including predicted pixel type, predicted connection type between pixels, and predicted rotation angle), calculate the difference (e.g., loss) between the predicted feature data and the supervisory information, and adjust the network parameters of the initial recognition model based on the difference.

[0073] In one embodiment, the number of sample text images can be multiple, and the multiple sample text images can include images of only one language category (for example, only target language text images), and the server trains the model based on the images of this language category; the multiple sample text images can also include images of multiple language categories (for example, target language text images and Japanese text images), and the server trains the model based on images of multiple language categories. During the training process, the server can mix the corpus of each language in proportion, for example, mix the training corpus of each language with reference to the language distribution ratio of online text images, so that the text recognition model has better text recognition capabilities in actual business.

[0074] It should be noted that the aforementioned multiple language categories are all agglutinative languages. Since different languages ​​that are agglutinative languages ​​share similarities, pre-training the model using sample text images from multiple language categories can improve the model's usage scenarios and the model's accuracy in recognizing text in agglutinative language text images. During the aforementioned training process using sample text images from multiple language categories, the weights assigned to each language significantly influence the bias of the final model. To ensure that the final model is not biased towards any one language, the language weights can be evenly distributed. This method yields better training results when the amount of training data is comparable, but in actual business, the training corpus for each language varies significantly. Therefore, the language weights need to be adjusted based on the number of languages, the actual training data, and business needs.

[0075] In one embodiment, the model training method can be: using the loss function for back propagation to update the weights of each node in the initial recognition model, and when the loss function meets the preset conditions, determining the final weights of each node in the initial recognition model, wherein the loss function can use one or more of the 0-1 loss function, absolute value loss function, mean square error loss function, logarithmic loss function, cross entropy loss function and exponential loss function.

[0076] In one embodiment, the server can use the same training data set to perform segmented training on the initial detection model and the initial recognition model to obtain a text detection model and a text recognition model (for example, the initial detection model is trained using the first training data set to obtain a trained text detection model; the trained text detection model is then used to process the first training data set again, and the processed first training data set is used as training data for the initial recognition model, and finally a trained text recognition model is obtained); the server can also use the same training data set to jointly train the initial detection model and the initial recognition model to obtain a text detection model and a text recognition model.

[0077] It should be noted that joint training is a method for improving model prediction accuracy, which is also affected by many factors, such as the quality of the training dataset. Therefore, when using the method provided in this application, the appropriate training method should be selected based on the actual situation and prediction results to achieve higher model prediction accuracy.

[0078] In one embodiment, the server can use different training data sets to perform segmented training on the initial detection model and the initial recognition model respectively to obtain a text detection model and a text recognition model (for example, the initial detection model is trained using the first training data set to obtain a trained text detection model; and then the initial recognition model is trained using the second training data set to obtain a trained text recognition model); the server can also use different training data sets to jointly train the initial detection model and the initial recognition model to obtain a text detection model and a text recognition model.

[0079] In one embodiment, the text recognition model includes character decoding rules. In the above step S203, the text recognition model is called to perform text recognition on at least one first text area image to obtain the target text in the original image, which can be achieved according to the following steps.

[0080] (a1) Calling a text recognition model to perform text recognition on at least one first text region image to obtain a prediction probability distribution corresponding to the first text region image, wherein the prediction probability distribution includes a character code prediction probability for each position in the first text region image.

[0081] In an embodiment of the present application, the server divides the original image into at least one first text area image based on the idea of ​​dividing by text lines. Each first text area image can correspond to an area image corresponding to a line of text divided by text lines in the original image. The text line includes multiple positions, each position can correspond to a character, and the predicted probability distribution corresponding to the first text area image includes the character encoding prediction probability corresponding to each position in the text line. The character encoding refers to the digital code obtained after the character is encoded and converted according to the character encoding rules. The character encoding prediction probability refers to the probability distribution of each position in the text line corresponding to multiple preset digital codes.

[0082] For example, assuming that there are only four character codes [1, 2, 3, 4] in a character code library, and the text line where the first text area image is located includes three positions (corresponding to three characters), then the predicted probability distribution corresponding to the first position in the first text area image can be in the form of [(1 / 10), (3 / 10), (4 / 10), (2 / 10)]. The above predicted probability distribution indicates that the probability of this position being character code 1 is 1 / 10, the probability of being character code 2 is 3 / 10, the probability of being character code 3 is 4 / 10, and the probability of being character code 4 is 2 / 10. The above only lists the probability distribution of a certain position in the text line, and its predicted probability distribution can be in the form of an array. For the entire text line, the predicted probability distribution can be a matrix (each position corresponds to a row of data in the matrix, that is, an array).

[0083] (a2) Determine a third character code prediction sequence corresponding to the first text region image based on the character code prediction probability of each position.

[0084] In one embodiment, the character code with the highest probability in the probability distribution corresponding to each position in the text line can be used as the predicted character code for that position. The predicted character codes for all positions in the text line can be obtained by the above method, that is, the third character code prediction sequence corresponding to the first text area image is obtained. For example, the predicted probability distribution corresponding to the first position in the first text area image is [(1 / 10), (3 / 10), (4 / 10), (2 / 10)], then the predicted character code corresponding to that position is the character code 3 corresponding to the probability 4 / 10. The third character code prediction sequence can be in the form of [3, 2, 1, 4], where each number represents the predicted character code corresponding to that position in the text line.

[0085] (a3) determining the target text in the original text image according to the third character code prediction sequence and the character decoding rule.

[0086] In an embodiment of the present application, each character code in the third character code prediction sequence corresponds to a unique character in the target language. By restoring the third character code prediction sequence through character decoding rules, a sequence composed of characters in the corresponding target language can be obtained, which is the target text.

[0087] For example, the target language includes characters [α, β, γ, δ, ε], and the third character code prediction sequence may include all numeric codes [0, 1, 2, 3, 4]. The character decoding rule stores the correspondence between the target language text and the numeric codes of the characters in the third character code prediction sequence [α-0, β-1, γ-2, δ-3, ε-4]. Based on this correspondence, the third character code prediction sequence can be converted into the corresponding target text. For example, the third character code prediction sequence is [0, 3, 1, 3, 2], and the corresponding target text is [αδβδγ].

[0088] The above method utilizes pre-defined character encoding rules to enhance the ability of character sequences to represent rich data. This method can be well-suited for text recognition in languages ​​with variant forms (e.g., the target language). By converting numeric codes using character encoding rules, it can flexibly distinguish and represent characters with different variant forms.

[0089] In one embodiment, the characters in the first text region image are characters corresponding to a target language. The target language includes multiple semantic characters, and each semantic character corresponds to multiple deformed characters. The deformed characters are selected from the group consisting of an independent character, a pre-concatenated character, a double-concatenated character, and a post-concatenated character. The deformed character type is determined based on the position of the semantic character in the target language text.

[0090] The target language will be taken as an example to explain the above-mentioned character encoding rules and character decoding rules in detail. The target language is an agglutinative language with a total of 33 basic letters (i.e. semantic characters), and each basic letter corresponds to a variety of modified letters (i.e. modified characters). When the target language letters form a vocabulary, different modified forms need to be selected due to their different positions in the word. Since the Unicode encoding (Unicode) of the target language is discontinuous in the Arabic character encoding, unlike Chinese, the target language has a Unicode encoding in an extended area. The target language has a total of 33 letters, and each semantic letter has a writing method, which is correspondingly encoded in the basic area. In the selection, except for a few semantic letters, other semantic letters have four modified forms, which are correspondingly encoded in the extended area. This application first establishes the correspondence between the basic area and the extended area, and digitally encodes all semantic letters; then formulates a selection rule framework (including character encoding rules and character decoding rules), and selects different modified letters for connection according to the different positions of the target language semantic letters in the word. From the perspective of language rules, selection rules are formulated for the random combination of 33 basic letters of the target language and the corresponding 105 inflected forms.

[0091] In one embodiment, the character encoding rules are used to convert the text encoding of the text into a first character encoding sequence, and to convert the first character encoding sequence into a second character encoding sequence, the first character encoding sequence includes the encoding of the semantic characters, and the second character encoding sequence includes the encoding of the deformed characters corresponding to the semantic characters, so as to distinguish different character deformation forms, and the character decoding rules include a mapping relationship between the text encoding and the character encoding sequence, which is used to restore the second character encoding sequence to text.

[0092] In the above-mentioned target language text recognition scenario, the text encoding may be Unicode encoding, and the third character encoding prediction sequence may include modified letters (ie, modified characters) of basic letters in the target language.

[0093] In one embodiment, assuming that there are 33 semantic characters and 105 corresponding variant characters in the target language, each semantic letter can correspond to a variety of variant characters according to its position in the word of the target language text (e.g., independent existence, before the word, in the word, at the end of the word). Then, each of the 33 semantic characters can be mapped to a unique first digital code, so that different semantic characters can be distinguished by the first digital code. For example, the 33 semantic characters are mapped to numbers starting from 0 in order. At this time, the character encoding sequence corresponding to the text composed of these 33 semantic characters arranged in sequence is [0, 1, 2, 3, 4, 5, 6, ..., 32]. Each semantic character can correspond to multiple variant characters (for example, a semantic character with a character code of 0 can correspond to 4 variant characters. In this case, 4 second digital codes are required to distinguish these 4 variant characters. A total of 33 semantic characters can correspond to 105 variant characters. In this case, 105 second digital codes are required to distinguish these 105 variant characters). The 105 variant characters are mapped to numbers starting from 0 in order. At this time, the character code sequence corresponding to the 105 variant characters arranged in sequence is [0, 1, 2, 3, 4, 5, 6, ..., 104].

[0094] In one embodiment, the two character code sequences (e.g., the first character code sequence and the second character code sequence) may be two independent sets of digital codes that do not have the same digital code. For example, a text composed of 33 semantic characters arranged in sequence corresponds to the character code sequence [0, 1, 2, 3, 4, 5, 6, ..., 32]; 105 deformed characters are mapped to digital codes starting from 33, and the corresponding character code sequence when arranged in sequence is [33, 34, 35, 36, 37, 38, ..., 137].

[0095] It should be noted that the above-mentioned character encoding sequence format and mapping rules are only exemplary. The specific mapping method can be flexibly set according to actual business conditions such as the number of letters in the target language, and this application does not limit this.

[0096] In one embodiment, character encoding rules and character decoding rules can be used to perform forward and reverse encoding and decoding operations to verify the accuracy of the selection rule framework. Specifically, forward encoding (i.e., converting the target language letter Unicode code into the corresponding digital code using the character encoding rules) and reverse encoding (i.e., converting the digital code into the corresponding target language letter Unicode code using the character encoding rules) can be used.

[0097] In an embodiment of the present application, the initial recognition model includes multiple branches, each of which has a different neural network model. Among them, the first neural network can predict the second text area image to obtain a first character code prediction sequence; the second neural network can predict the second text area image to obtain a second character code prediction sequence. By obtaining feature information of different dimensions of the second text area image through different neural networks, and then fusing the results generated by different neural networks to obtain the final processing result, the processing capability of the text recognition model can be improved, thereby making full use of the feature information and obtaining more accurate recognition results.

[0098] In the embodiment of the present application, by combining the first neural network and the second neural network, the initial recognition model is trained based on a multi-task collaborative method to ensure good training results. After the initial recognition model training is completed, the first neural network and the second neural network both have strong text recognition capabilities. However, in the text recognition model inference stage (i.e., the application stage), using two neural networks at the same time will reduce the recognition efficiency. By using the more efficient first neural network branch to recognize the text content in the original image, the recognition efficiency is significantly improved while ensuring the accuracy of text recognition.

[0099] This application integrates character encoding rules and character decoding rules into the text recognition model. The character encoding rules can convert the character encoding sequence into another set of digital codes, and the character decoding rules can make each digital code in the digital code correspond one-to-one with all the characters in a certain character library (for example, all the characters included in a certain target language).

[0100] For example, the character encoding prediction sequence is [0, 3, 1, 3, 2], and the character encoding rule includes the corresponding relationship [0-51, 1-52, 2-53, 3-54, 4-55...]. Then, the character encoding prediction sequence [0, 3, 1, 3, 2] can be converted to [51, 54, 52, 54, 53] according to the character encoding rule. If the character decoding rule includes the corresponding relationship between a certain language text and the character encoding numeric code [51-α, 52-β, 53-γ, 54-δ, 55-ε...], then [51, 54, 52, 54, 53] can be further converted to the text [αδβδγ].

[0101] The beneficial effects of the present application are as follows: the present application obtains an original image containing text content and performs segmentation processing to obtain at least one first text area image, and then processes the single first text area image through a text recognition model to obtain the target text in the original image, thereby avoiding the mutual influence between multiple lines of text and improving the accuracy of image processing. The text recognition model in the present application is obtained by training an initial recognition model including a first neural network and a second neural network. By processing the first text area image through two neural networks with different network structures, the feature information of the image can be fully utilized, the processing capability of the model can be improved, and thus the accuracy of text recognition can be improved.

[0102] During model training, the reference text corresponding to the sample image is mapped to a character encoding reference sequence using established character encoding rules, which serves as supervisory information for model training. These character encoding rules can flexibly transform the text corresponding to the target language according to the grammatical features of the target language, enabling the converted character encoding reference sequence to flexibly distinguish and represent characters with different inflected forms. This method is well-suited for text recognition in languages ​​with inflected forms, such as agglutinative languages. By integrating character encoding rules into text recognition and utilizing a text recognition model to process the corresponding character encoding reference sequence, the accuracy of recognition results for agglutinative language text can be improved.

[0103] The present application proposes that during the training process of the text detection model and the text recognition model, the server can mix corpora of multiple languages ​​such as agglutinative language in proportion, so that the text recognition ability of the text recognition model in actual business is better, the usage scenarios of the model are improved, and the accuracy of the model in recognizing text in agglutinative language text images is improved. The present application proposes that the server can use the same training data set to perform segmented training on the initial detection model and the initial recognition model; or use the same training data set to perform joint training on the initial detection model and the initial recognition model; or use different training data sets to perform segmented training or joint training on the initial detection model and the initial recognition model, and select the appropriate training method according to the actual business situation to achieve higher model prediction accuracy. The diversified training methods also improve the flexibility of model training. After the text detection model is trained using the sample data set, the sample data set can be processed by the text detection model as training data for the text recognition model, thereby improving the utilization rate of the data.

[0104] This application proposes that, in the context of target language text, a correspondence between the basic area and the extended area in the target language text can be established, and all semantic letters can be digitally encoded. A selection rule framework (including character encoding rules and character decoding rules) is then formulated to select different deformed letters for connection based on the position of the target language semantic letters in the word. This approach can be used to perform target language text recognition tasks, effectively solving the problem of low recognition accuracy caused by deformed characters.

[0105] See also Figure 3 , Figure 3 This is a flowchart of a text recognition method provided by an exemplary embodiment of the present application, in which the method is applied to a server (referring to Figure 1D Taking the server 102 in FIG. 1 as an example, the method may include the following steps:

[0106] S301 : Acquire a sample image and second supervisory information, where the second supervisory information includes one or more of a reference pixel type, a reference connection type between pixels, and a reference rotation angle of a third text region image in the sample image.

[0107] In the embodiment of the present application, during the application of the text detection model, the first text region image is obtained by processing the original image using the text detection model. The text detection model is obtained by pre-training an initial detection model using sample images. Steps S301-S303 describe the specific process of training the initial detection model to obtain the text detection model. The main function of the text detection model is to process sample images containing text content to obtain a text region image corresponding to each line of text.

[0108] Among them, the third text area image is the area image obtained after dividing the sample image according to the text line level. The number of third text area images can be one or more, corresponding to one or more lines of text in the sample image. The initial detection model can use a preset angle to take a line of text in the sample image as the third text area image. For example, if the preset angle is 0 degrees, then the image area where the text line is located in the horizontal direction can be used as the third text area image; if the preset angle is 30 degrees, then the image area where the text line is located in the 30-degree tilt direction can be used as the third text area image; if the preset angle is 90 degrees, then the image area where the text line is located in the vertical direction can be used as the third text area image. The above process of obtaining the third text area image from the sample image can also be automatically recognized by the initial detection model. For example, the initial detection model automatically recognizes the tilt angle of each line of text in the sample image and the length and width of the text line, and then generates the corresponding third text area image.

[0109] The second supervisory information is the annotation information of the sample image, and the second supervisory information includes one or more of the reference pixel type, reference connection type between pixels, and reference rotation angle of each third text area image in the sample image. The reference pixel type can supervise the predicted pixel type for predicting text areas and non-text areas; the reference connection type between pixels can supervise the predicted connection type for predicting the connection relationship between pixels; the reference rotation angle can supervise the predicted rotation angle for predicting the rotation angle of characters in the third area image. The above-mentioned supervisory information can be used to perform multi-dimensional supervision on the training of the text detection model, improve the processing capability of the text detection model, and thus improve the accuracy of the processing results of the text detection model.

[0110] In the following embodiment, model training will be described by taking one of the third text region images in the sample images as an example.

[0111] S302: Calling the initial detection model to process the third text region image to obtain predicted feature data of the third text region image, where the predicted feature data includes one or more of predicted pixel type, predicted connection type between pixels, and predicted rotation angle.

[0112] In an embodiment of the present application, the number of types of prediction feature data can be consistent with the number of types of prediction feature data, and the server can perform model training on the initial detection model using one or more prediction feature data and their corresponding prediction feature data.

[0113] In one embodiment, when the predicted feature data includes three types: predicted pixel type, predicted connection type between pixels, and predicted rotation angle, the above-mentioned calling of the initial detection model to process the third text area image to obtain the predicted feature data of the third text area image can be implemented according to the following steps.

[0114] (b1) Calling the initial detection model to perform text detection on the third text area image to obtain the text area and non-text area in the third text area image, and determining the predicted pixel type of each pixel according to the position, text area and non-text area of ​​each pixel in the third text area image.

[0115] In an embodiment of the present application, the initial detection model can obtain the area where the text is located in the third text area image (or the set of pixels where the text is located) through image segmentation, text content recognition and other methods, thereby dividing the third text area image into text area and non-text area; then match the text area and non-text area through the position of each pixel, and finally obtain the predicted pixel type of each pixel.

[0116] In one embodiment, the predicted pixel type of each pixel may include positive pixels and negative pixels. The initial detection model may determine the pixel prediction type of the pixels corresponding to the text area as positive pixels, and determine the pixel prediction type of the pixels corresponding to the non-text area as negative pixels. The pixels in the overlapping area of ​​multiple texts may also be determined as negative pixels.

[0117] (b2) Calling the initial detection model to process the predicted pixel types of two pixels included in each adjacent pixel pair in the third text region image to obtain the corresponding predicted connection type between the pixels.

[0118] In the embodiment of the present application, there are multiple pixels in the third text area image, each pixel and its eight adjacent pixels can form eight adjacent pixel pairs, and the predicted connection type between pixels represents the pixel type relationship between two pixels in the adjacent pixel pair.

[0119] In one embodiment, the predicted connection types between pixels may include positive connections and negative connections. The initial detection model may determine the predicted connection type of an adjacent pixel pair in which the pixel prediction types of both pixels are negative pixels as a negative connection; determine the predicted connection type of an adjacent pixel pair in which the pixel prediction types of both pixels are positive pixels as a positive connection; and determine the predicted connection type of an adjacent pixel pair in which the pixel prediction type of one pixel is a negative pixel and the pixel prediction type of one pixel is a positive pixel as a positive connection.

[0120] To predict pixel types and predicted connection types between pixels, the server can set two different thresholds to obtain a set of positive pixels and a set of positive connections. Then, using positive connections, the positive pixels are grouped together to form a connected domain set, where each element represents a text instance in the image. Because errors can occur during text region recognition, causing pixels that originally belong to text regions to be identified as non-text regions, and because text typically appears in blocks in images, this method accurately captures the text region images to be identified, improving the accuracy of subsequent text detection processing based on the text region images.

[0121] It should be noted that the above configuration methods of predicted pixel types and predicted connection types between pixels are only exemplary and can be flexibly set according to actual business conditions. This application does not impose any restrictions on this.

[0122] (b3) Calling the initial detection model to determine the predicted rotation angle of the characters in the third text region image.

[0123] In the embodiment of the present application, the initial detection model can automatically identify the tilt angle of the characters in the sample image and obtain the predicted rotation angle. The tilt angle is obtained by calculating the offset angle of the upper and lower lines of the horizontal text box (the text line in the image) relative to the horizontal axis.

[0124] In one embodiment, the initial detection model can predict the rotation angle of each character in the third text area image separately to obtain the rotation angle corresponding to each character, and then calculate the mean of the rotation angle corresponding to each character to obtain the predicted rotation angle; the initial detection model can also regard all the characters in the third text area image as a whole, and then predict the rotation angle of the whole to obtain the predicted rotation angle.

[0125] S303: Adjust parameters of the initial detection model based on the second supervisory information and the predicted feature data to obtain a text detection model.

[0126] In an embodiment of the present application, the predicted feature data includes the predicted pixel type, the predicted connection type between pixels and the predicted rotation angle, and the second supervision information includes the reference pixel type, the reference connection type between pixels and the reference rotation angle.

[0127] The server can calculate pixel type differences by comparing the predicted pixel type with the reference pixel type; calculate connection type differences by comparing the predicted connection type with the reference connection type; and calculate rotation angle differences by comparing the predicted rotation angle with the reference rotation angle. The server then adjusts the parameters of the initial detection model based on the differences in pixel type, connection type, and rotation angle to produce a text detection model. This supervision information allows for multi-dimensional supervision of the text detection model's training, improving its processing capabilities and, consequently, the accuracy of its processing results.

[0128] The difference may be calculated by calculating the loss through a loss function, and the loss function may use one or more of a cross entropy loss function, a 0-1 loss function, an absolute value loss function, a mean square error loss function, a logarithmic loss function, and an exponential loss function.

[0129] S304: Calling a text detection model to process the sample image to obtain at least one second text region image corresponding to the sample image.

[0130] In the embodiment of the present application, during the application of the text recognition model, the target text is obtained by processing the first text region image using the text recognition model. The text recognition model requires pre-training the initial recognition model using sample data (e.g., the second text region image). Steps S304-S307 are the specific process of training the initial recognition model to obtain the text recognition model. The main function of the text recognition model is to perform text recognition processing on the text region image corresponding to each line of text to obtain the final text recognition result.

[0131] S305 : Acquire a character code reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image.

[0132] In the embodiment of the present application, the second text region image is obtained from a sample image, and the sample image includes corresponding reference text, i.e., annotated text. Therefore, the second text region image also has corresponding reference text. The character encoding reference sequence is obtained by character mapping the reference text corresponding to the second text region image using character encoding rules.

[0133] In one embodiment, the above-mentioned obtaining of a character encoding reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image may be implemented according to the following steps.

[0134] (c1) Obtain the reference text corresponding to the second text region image.

[0135] In one embodiment, the reference text corresponding to the second text region image may be extracted from the reference text corresponding to the sample image, or may be annotation information separately annotated for the second text region image.

[0136] (c2) performing a first encoding process on the reference text corresponding to the second text region image to obtain a text code.

[0137] In one embodiment, the initial recognition model may perform a first encoding process on the reference text to obtain a digital code type that can be processed by the computer. For example, the reference text may be converted into a corresponding Unicode format using Unicode encoding. Unicode was created to address the limitations of traditional character encoding schemes. It sets a unified and unique binary code for each character in each language to meet the requirements of cross-language and cross-platform text conversion and processing.

[0138] (c3) performing a second encoding process on the text encoding based on the character encoding rule to obtain a character encoding reference sequence corresponding to the second text region image.

[0139] In an embodiment of the present application, a text encoding (e.g., Unicode encoding) can be converted into a set of character encoding sequences, i.e., a character encoding reference sequence, through character encoding rules. Through the above-mentioned character encoding rules, flexible conversion between text encoding and character encoding sequences is achieved, and the ability of character sequences to represent rich data is improved. The above-mentioned method can be well applied to text recognition processing in languages ​​with variant forms, by converting digital codes through character encoding rules, and flexibly distinguishing and representing characters with different variant forms.

[0140] S306 : Call the first neural network and the second neural network included in the initial recognition model to process the second text region image respectively to obtain a first character code prediction sequence and a second character code prediction sequence.

[0141] In an embodiment of the present application, the initial recognition model includes multiple branches, each of which has a different neural network model. Among them, the first neural network can predict the second text area image to obtain a first character code prediction sequence; the second neural network can predict the second text area image to obtain a second character code prediction sequence. By obtaining feature information of different dimensions of the second text area image through different neural networks, and then fusing the results generated by different neural networks to obtain the final processing result, the processing capability of the text recognition model can be improved, thereby making full use of the feature information and obtaining more accurate recognition results.

[0142] S307 : Based on the character code reference sequence, the first character code prediction sequence, and the second character code prediction sequence, parameters of the first neural network and the second neural network included in the initial recognition model are adjusted to obtain a text recognition model.

[0143] In one embodiment, the above-mentioned parameter adjustment of the first neural network and the second neural network included in the initial recognition model based on the character encoding reference sequence, the first character encoding prediction sequence and the second character encoding prediction sequence to obtain the text recognition model can be implemented according to the following steps.

[0144] (d1), determining a first difference between the character encoding reference sequence and the first character encoding prediction sequence, and a second difference between the character encoding reference sequence and the second character encoding prediction sequence.

[0145] (d2) Adjusting the network parameters of the first neural network and the second neural network included in the initial recognition model according to the first difference and the second difference.

[0146] The difference may be calculated by calculating the loss through a loss function, and the loss function may use one or more of a cross entropy loss function, a 0-1 loss function, an absolute value loss function, a mean square error loss function, a logarithmic loss function, and an exponential loss function.

[0147] (d3) The first neural network after parameter adjustment is used as a text recognition model, and the character decoding rules are embedded in the text recognition model.

[0148] In the embodiment of the present application, by combining the first neural network and the second neural network, the initial recognition model is trained based on a multi-task collaborative method to ensure good training results. After the initial recognition model training is completed, the first neural network and the second neural network both have strong text recognition capabilities. However, in the text recognition model inference stage (i.e., the application stage), using two neural networks at the same time will reduce recognition efficiency. By using the more efficient first neural network to recognize the text content in the original image, the recognition efficiency is significantly improved while ensuring the accuracy of text recognition.

[0149] In one embodiment, the first neural network may include a continuous temporal classification network, and the second neural network may include a bidirectional long short-term memory network and an attention mechanism network.

[0150] S308: Obtain the original image.

[0151] S309: Segment the original image to obtain at least one first text region image.

[0152] S310. Call a text recognition model to perform text recognition on at least one first text area image to obtain the target text in the original image. The text recognition model is obtained by training the first neural network and the second neural network included in the initial recognition model based on the sample image and the first supervisory information. The first supervisory information includes a character encoding reference sequence obtained by processing the reference text corresponding to the sample image based on the character encoding rule.

[0153] In the embodiment of the present application, the server can perform text recognition on each of the at least one first text region images by calling a text recognition model to obtain the target text in the original image. The text recognition model is trained using sample images (including corresponding reference text, i.e., annotated text).

[0154] During text recognition model training, the server processes sample images using the initial recognition model to generate predicted text. The server then adjusts the network parameters of the initial recognition model based on the predicted text and first supervisory information to generate the text recognition model. The first supervisory information includes a character encoding reference sequence, which is obtained by mapping the reference text corresponding to the sample image using character encoding rules.

[0155] Through the above method, the character encoding reference sequence is used as supervisory information for the text recognition model and combined with the predicted text to complete the training of the text recognition model. Through the above method, when a character has multiple deformation forms, the character can be mapped to different deformation mapping codes according to the character mapping rules to distinguish the deformation forms in different situations.

[0156] In one embodiment, each sample text image may include a text image, annotated text, and other supervisory information (i.e., annotation information, such as reference pixel type, reference connection type between pixels, and reference rotation angle, etc.). For example, the server may process the first text region image using an initial recognition model to obtain predicted feature data (including predicted pixel type, predicted connection type between pixels, and predicted rotation angle), calculate the difference (e.g., loss) between the predicted feature data and the supervisory information, and adjust the network parameters of the initial recognition model based on the difference.

[0157] The specific implementation of steps S308 to S310 refers to the relevant description of steps S201 to S203 in the above embodiment, which will not be repeated here.

[0158] Based on the method provided in the above embodiments, this application proposes a text detection and recognition model based on multi-task collaborative training. Figure 4A , this figure is a flow chart of the model training process of the text detection and recognition model, which includes two sub-models, namely the text detection model and the text recognition model.

[0159] The text detection model is obtained by pre-training the initial detection model. The training process of the initial detection model is as follows: taking the target language text as an example, first input a sample image (such as a picture in the target language scene) and use a convolutional neural network (such as the VGG-16 network. In the structure shown in the figure, the last fully connected layer of the convolutional neural network is replaced with a convolutional layer. VGG has 16 layers, including 13 convolutional layers and 3 fully connected layers. The main work of this model is to prove that increasing the depth of the network can affect the final performance of the network to a certain extent).

[0160] Convolutional neural networks are used to extract visual features for feature fusion and pixel prediction. The method of feature fusion and pixel prediction is based on the FPN idea (feature pyramid network), that is, the size of the convolution layer is halved, but the number of convolution kernels is doubled). The extracted visual features are then input into three independent feature prediction modules: one for text / non-text prediction, one for prediction of the connection relationship between pixels, and one for prediction of rotation angle. All three can use the Softmax loss function, and output 1x2=2 channels (i.e., corresponding to text / non-text classification), 8x2=16 channels (i.e., whether the 8 neighborhood directions of a given pixel are connected), and 1x1=1 channel (i.e., prediction of rotation angle).

[0161] The first branch is supervised by positive and negative pixels. This application stipulates that all pixels falling within the text area are marked as positive pixels, pixels outside the text area are marked as negative pixels, and multiple text overlapping areas are also marked as negative pixels.

[0162] Each link supervision information of the second branch is determined by two pixels in both directions. For a given pixel and its eight adjacent pixels, if both pixels are positive, the link between them is a positive connection; if one pixel is positive and the other is negative, the link between them is also a positive connection; if both pixels are negative, the link between them is a negative connection.

[0163] The third branch is supervised by the angle information of the text line. The angle is calculated by averaging the offset angles of the top and bottom lines of the horizontal text box relative to the horizontal axis. For the prediction results of the first two branches, two different thresholds are set to obtain a set of positive pixels and a set of positive connections. Positive connections are used to group positive pixels together, resulting in a set of connected domains. Each element in this connected domain set represents a text instance.

[0164] The above three branches can respectively obtain the predicted pixel type, predicted connection type between pixels and predicted rotation angle corresponding to each text area image in the sample image. Since the sample image includes text image, annotated text, and other supervisory information (i.e., annotation information, such as reference pixel type, reference connection type between pixels and reference rotation angle, etc.), the text / non-text loss can be obtained by predicting the pixel type and the reference pixel type, the eight-neighborhood connection loss can be obtained by predicting the connection type between pixels and the reference connection type between pixels, and the rotation angle loss can be obtained by predicting the rotation angle and the reference rotation angle. Using one or more of the text / non-text loss, the eight-neighborhood connection loss and the rotation angle loss, the network parameters of the initial detection model can be adjusted to obtain a text detection model.

[0165] The text recognition model is obtained by pre-training the initial recognition model. The training process of the initial recognition model is as follows: first, a text area image is obtained (the text area image can be obtained by processing a sample text image using a trained text detection model, or it can be a set of new data obtained); the basic network framework of the neural network model can be ResNet-34, and the last layer of convolutional feature map of ResNet-34 is divided into two branches. One branch is based on continuous temporal classification (Connectionist Temporal Classification, CTC) to obtain the predicted distribution probability (i.e., the first character code prediction probability), and the other branch is based on a bidirectional long short-term memory network (Long Short-Term Memory, LSTM) and an attention mechanism (Attention) to obtain the predicted distribution probability (i.e., the second character code prediction probability).

[0166] Since text region images include annotated text, the annotated text can be used to obtain supervisory information (i.e., character code reference sequence) for supervising the predicted distribution probability. The continuous temporal classification loss is derived from the first character code prediction probability and the character code reference sequence, while the attention mechanism loss is derived from the second character code prediction probability and the character code reference sequence. Using one or more of the continuous temporal classification loss and the attention mechanism loss, the network parameters of the initial recognition model can be adjusted to obtain a text recognition model.

[0167] Since LSTM performs string processing over time, that is, the prediction at the current moment depends on the historical information before the current moment. Therefore, it will seriously affect the computational efficiency of the network. In order to integrate the powerful sequence modeling capabilities of LSTM, the present invention uses an attention branch with LSTM to supervise the learning of the CTC branch during the training phase. In the inference phase, only the CTC branch is used to predict the distribution probability of characters. This method can not only use the bidirectional LSTM to learn the influence relationship between feature sequences, but also avoid the huge inference time consumption caused by serial calculations. In addition, the text recognition model can fully utilize all feature information before the current position through the encoding and decoding mechanism in the attention branch, and can also utilize the feature information after the current position through the method of calculating the global probability in the CTC branch, thereby making full use of the feature information and obtaining more accurate recognition results.

[0168] The following is an introduction to the selection rule framework proposed in this application. The selection rule framework mainly includes character encoding rules and character decoding rules. Figure 4B, this figure is a flow chart of the selection rule framework proposed in this application. First, input the target language text (the target language text can be the reference text corresponding to the sample image), and first convert the target language text into Unicode (i.e. text encoding) in the selection rule framework; then convert the text encoding into a first character encoding sequence through the character encoding rule; then read each number in the first character encoding sequence in a circular manner, and convert it according to the position of each number through the character encoding rule to obtain a second character encoding sequence; since each number that may exist in the second character encoding sequence uniquely corresponds to a character, it can be restored to the corresponding target language text through the character decoding sequence. It should be noted that in the character encoding rule, when recognizing text, 0 can be added at the beginning, end, and between words to separate words, thereby improving the feature expression ability of the recognized sequence.

[0169] Based on the character encoding and decoding rules in the above selection rule framework, this embodiment will introduce the application process of the text detection and recognition model based on multi-task collaborative training. Figure 4C , the figure is a flow chart of the application process of the text detection and recognition model. Taking the target language text as an example, the original image (such as a picture in the target language scene) is first input, and multiple text area images are obtained through the text detection model; for each text area image in the multiple text area images, the text recognition model is used to obtain the first character encoding sequence corresponding to the text area image according to the character encoding rules, and the second character encoding sequence is obtained according to the character encoding rules and the first character encoding sequence; the second character encoding sequence is restored to the corresponding target text through the character decoding rules in the selection rule framework.

[0170] This application proposes a model training method with better training effect, which trains the text detection model and the text recognition model in segments. First, the text detection model (the front-end detection model) is trained, and then the text recognition model (the back-end line-level text recognition model) is trained. SGD gradient optimization is used when training the text detection model, with momentum set to 0.9 and weight decay set to 5*10. -4 , randomly initialize the VGG network using the Xavier method, and set the learning rate to 10 for the first 100 iterations. -3 , the rest are fixed at 10 -2 The backend model uses the AdaDelta optimizer with a decay rate of 0.95, a gradient clipping amplitude of 5, and a learning rate that changes as the training progresses, lr = warmup 0.5 *min(s 0.5 ,s*warmup -1.5 ). Where s represents the current number of training steps, and warmup = 25000 controls the learning process to increase first and then decrease.

[0171] The beneficial effects of the present application are as follows: the present application proposes to perform multi-dimensional supervision on the training of the text detection model through supervisory information such as reference pixel type, reference connection type between pixels and reference rotation angle, which can improve the processing ability of the text detection model, thereby improving the accuracy of the processing results of the text detection model. For the predicted pixel type and the predicted connection type between pixels, the present application proposes to obtain a positive pixel set and a positive connection set by setting two different thresholds, and then use positive connections to group the positive pixels together to generate a connected domain set, and each element in the set represents a text instance in the image. Since errors may occur when identifying text areas, resulting in pixels that were originally text areas being identified as pixels of non-text areas, and text usually appears in blocks in images, the above method can accurately obtain the text area image that needs to be identified, thereby improving the accuracy of the subsequent processing results of text detection based on text area images.

[0172] This application also proposes that when performing loss calculation, one or more of the cross entropy loss function, 0-1 loss function, absolute value loss function, mean square error loss function, logarithmic loss function and exponential loss function can be used to improve the flexibility of loss calculation.

[0173] This application proposes a flexible conversion between text encoding and character encoding sequences through character encoding rules, enhancing the ability of character sequences to represent rich data. This method can be well applied to text recognition in languages ​​with variant forms. By converting digital codes through character encoding and character decoding rules, it can flexibly distinguish and represent characters with different variant forms.

[0174] The present application proposes that the initial recognition model includes multiple branches, each branch has a different neural network model. Among them, the first neural network can predict the second text area image to obtain a first character code prediction sequence; the second neural network can predict the second text area image to obtain a second character code prediction sequence. By obtaining the feature information of different dimensions of the second text area image through different neural networks, and then fusing the results generated by different neural networks to obtain the final processing result, the processing capability of the text recognition model can be improved, thereby making full use of the feature information and obtaining more accurate recognition results. The present application proposes to adopt a multi-neural network joint training method in the training stage of the text recognition model to improve the training effect. However, in the inference stage of the text recognition model (i.e., the application stage), the simultaneous use of two neural networks will reduce the recognition efficiency. By using the more efficient first neural network branch to recognize the text content in the original image, the recognition efficiency is significantly improved while ensuring the accuracy of text recognition.

[0175] When the present application is applied to the actual target language text recognition scenario, the first neural network included in the text recognition model in the present application can be a continuous temporal classification network (CTC), and the second neural network can be a combination of a bidirectional long short-term memory network (LSTM) and an attention mechanism network (Attention). Since LSTM performs string processing over time, that is, the prediction at the current moment depends on the historical information before the current moment. Therefore, it will seriously affect the computational efficiency of the network. In order to incorporate the powerful sequence modeling capabilities of LSTM, the present application uses an attention branch with LSTM to supervise the learning of the CTC branch during the training phase. In the inference phase, only the CTC branch is used to predict the distribution probability of characters. This method can not only use bidirectional LSTM to learn the influence relationship between feature sequences, but also avoid the huge inference time consumption caused by serial calculations. In addition, the text recognition model can fully utilize all feature information before the current position through the encoding and decoding mechanism in the attention branch, and can also utilize the feature information after the current position through the method of calculating the global probability in the CTC branch, thereby making full use of the feature information and obtaining more accurate recognition results.

[0176] See also Figure 5 , Figure 5 : is a schematic block diagram of a text recognition device provided in an embodiment of the present application. The text recognition device may specifically include:

[0177] An acquisition module 501 is used to acquire an original image;

[0178] The processing module 502 is configured to segment the original image to obtain at least one first text region image.

[0179] The recognition module 503 is used to call a text recognition model to perform text recognition on the at least one first text area image to obtain the target text in the original image. The text recognition model is obtained by training the first neural network and the second neural network included in the initial recognition model based on the sample image and the first supervision information. The first supervision information includes a character encoding reference sequence obtained by processing the reference text corresponding to the sample image based on the character encoding rules.

[0180] Optionally, the processing module 502 is further configured to:

[0181] Calling a text detection model to process the sample image to obtain at least one second text region image corresponding to the sample image;

[0182] obtaining a character encoding reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image;

[0183] Invoking the first neural network and the second neural network included in the initial recognition model to process the second text region image respectively to obtain a first character code prediction sequence and a second character code prediction sequence;

[0184] Based on the above-mentioned character code reference sequence, the above-mentioned first character code prediction sequence and the above-mentioned second character code prediction sequence, the parameters of the first neural network and the second neural network included in the above-mentioned initial recognition model are adjusted to obtain a text recognition model.

[0185] Optionally, when the processing module 502 is used to adjust parameters of the first neural network and the second neural network included in the initial recognition model based on the character code reference sequence, the first character code prediction sequence, and the second character code prediction sequence to obtain a text recognition model, it is specifically used to:

[0186] determining a first difference between the character code reference sequence and the first character code prediction sequence, and a second difference between the character code reference sequence and the second character code prediction sequence;

[0187] Adjusting network parameters of the first neural network and the second neural network included in the initial recognition model according to the first difference and the second difference;

[0188] The first neural network after parameter adjustment is used as a text recognition model, and the character decoding rules are embedded in the above text recognition model; wherein the above first neural network includes a continuous time series classification network, and the above second neural network includes a bidirectional long short-term memory network and an attention mechanism network.

[0189] Optionally, when the processing module 502 is used to obtain a character code reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image, it is specifically configured to:

[0190] Obtaining a reference text corresponding to the second text region image;

[0191] performing a first encoding process on the reference text corresponding to the second text region image to obtain a text code;

[0192] A second encoding process is performed on the text encoding based on a character encoding rule to obtain a character encoding reference sequence corresponding to the second text region image.

[0193] Optionally, the processing module 502 is further configured to:

[0194] Acquire a sample image and second supervisory information, where the second supervisory information includes one or more of a reference pixel type, a reference connection type between pixels, and a reference rotation angle of the third text region image in the sample image;

[0195] Invoking the initial detection model to process the third text region image to obtain predicted feature data of the third text region image, wherein the predicted feature data includes one or more of a predicted pixel type, a predicted connection type between pixels, and a predicted rotation angle;

[0196] Based on the second supervision information and the predicted feature data, the parameters of the initial detection model are adjusted to obtain a text detection model.

[0197] Optionally, the predicted feature data includes the predicted pixel type, the predicted connection type between pixels, and the predicted rotation angle. When the processing module 502 is used to call the initial detection model to process the third text region image to obtain the predicted feature data of the third text region image, it is specifically used to:

[0198] Calling the initial detection model to perform text detection on the third text region image to obtain a text region and a non-text region in the third text region image, and determining a predicted pixel type for each pixel based on a position of each pixel in the third text region image, the text region, and the non-text region;

[0199] Calling the initial detection model to process the predicted pixel types of two pixels included in each adjacent pixel pair in the third text region image to obtain the corresponding predicted connection type between the pixels;

[0200] The initial detection model is called to determine the predicted rotation angle of the characters in the third text region image.

[0201] Optionally, the text recognition model includes character decoding rules. When the recognition module 503 is used to call the text recognition model to perform text recognition on the at least one first text region image to obtain the target text in the original image, it is specifically used to:

[0202] Calling a text recognition model to perform text recognition on the at least one first text region image to obtain a prediction probability distribution corresponding to the first text region image, the prediction probability distribution including a character code prediction probability at each position in the first text region image;

[0203] Determining a third character code prediction sequence corresponding to the first text region image according to the character code prediction probabilities of the respective positions;

[0204] Determine a character decoding prediction sequence based on the third character encoding prediction sequence and the character decoding rule;

[0205] The target text in the original text image is determined based on the character decoding prediction sequence.

[0206] Optionally, the characters in the first text area image are characters corresponding to the target language, the target language includes multiple semantic characters, and each semantic character corresponds to multiple deformed characters; the character encoding rule includes a mapping relationship between text encoding and character encoding sequence, the character decoding rule includes a mapping relationship between character encoding sequence and character decoding sequence, the character encoding sequence includes the encoding of the semantic characters, and the character decoding sequence includes the encoding of the deformed characters.

[0207] It should be noted that the functions of the various functional modules of the text recognition device in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment and will not be repeated here.

[0208] See also Figure 6 , Figure 6 6 is a schematic block diagram of a computer device provided in an embodiment of the present application. As shown in the figure, the intelligent terminal in this embodiment may include: a processor 601, a storage device 602, and a communication interface 603. The processor 601, the storage device 602, and the communication interface 603 can exchange data.

[0209] The above-mentioned storage device 602 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage device 602 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a solid-state drive (SSD), etc.; the above-mentioned storage device 602 may also include a combination of the above-mentioned types of memory.

[0210] The processor 601 may be a central processing unit (CPU). In one embodiment, the processor 601 may also be a graphics processing unit (GPU). The processor 601 may also be a combination of a CPU and a GPU. In one embodiment, the storage device 602 is used to store program instructions, and the processor 601 may call the program instructions to perform the following operations:

[0211] Get the original image;

[0212] Segmenting the original image to obtain at least one first text region image;

[0213] A text recognition model is called to perform text recognition on the at least one first text area image to obtain the target text in the original image. The text recognition model is obtained by training the first neural network and the second neural network included in the initial recognition model based on the sample image and the first supervisory information. The first supervisory information includes a character encoding reference sequence obtained by processing the reference text corresponding to the sample image based on the character encoding rules.

[0214] Optionally, the processor 601 is further configured to:

[0215] Calling a text detection model to process the sample image to obtain at least one second text region image corresponding to the sample image;

[0216] obtaining a character encoding reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image;

[0217] Invoking the first neural network and the second neural network included in the initial recognition model to process the second text region image respectively to obtain a first character code prediction sequence and a second character code prediction sequence;

[0218] Based on the above-mentioned character code reference sequence, the above-mentioned first character code prediction sequence and the above-mentioned second character code prediction sequence, the parameters of the first neural network and the second neural network included in the above-mentioned initial recognition model are adjusted to obtain a text recognition model.

[0219] Optionally, when the processor 601 is used to adjust parameters of the first neural network and the second neural network included in the initial recognition model based on the character code reference sequence, the first character code prediction sequence, and the second character code prediction sequence to obtain a text recognition model, the processor 601 is specifically used to:

[0220] determining a first difference between the character code reference sequence and the first character code prediction sequence, and a second difference between the character code reference sequence and the second character code prediction sequence;

[0221] Adjusting network parameters of the first neural network and the second neural network included in the initial recognition model according to the first difference and the second difference;

[0222] The first neural network after parameter adjustment is used as a text recognition model, and the character decoding rules are embedded in the above text recognition model; wherein the above first neural network includes a continuous time series classification network, and the above second neural network includes a bidirectional long short-term memory network and an attention mechanism network.

[0223] Optionally, when the processor 601 is configured to obtain a character encoding reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image, the processor 601 is specifically configured to:

[0224] Obtaining a reference text corresponding to the second text region image;

[0225] performing a first encoding process on the reference text corresponding to the second text region image to obtain a text code;

[0226] A second encoding process is performed on the text encoding based on a character encoding rule to obtain a character encoding reference sequence corresponding to the second text region image.

[0227] Optionally, the processor 601 is further configured to:

[0228] Acquire a sample image and second supervisory information, where the second supervisory information includes one or more of a reference pixel type, a reference connection type between pixels, and a reference rotation angle of the third text region image in the sample image;

[0229] Invoking the initial detection model to process the third text region image to obtain predicted feature data of the third text region image, wherein the predicted feature data includes one or more of a predicted pixel type, a predicted connection type between pixels, and a predicted rotation angle;

[0230] Based on the second supervision information and the predicted feature data, the parameters of the initial detection model are adjusted to obtain a text detection model.

[0231] Optionally, the predicted feature data includes the predicted pixel type, the predicted connection type between the pixels, and the predicted rotation angle. When the processor 601 is used to call the initial detection model to process the third text region image to obtain the predicted feature data of the third text region image, it is specifically used to:

[0232] Calling the initial detection model to perform text detection on the third text region image to obtain a text region and a non-text region in the third text region image, and determining a predicted pixel type for each pixel based on a position of each pixel in the third text region image, the text region, and the non-text region;

[0233] Calling the initial detection model to process the predicted pixel types of two pixels included in each adjacent pixel pair in the third text region image to obtain the corresponding predicted connection type between the pixels;

[0234] The initial detection model is called to determine the predicted rotation angle of the characters in the third text region image.

[0235] Optionally, the text recognition model includes character decoding rules. When the processor 601 is used to call the text recognition model to perform text recognition on the at least one first text region image to obtain the target text in the original image, it is specifically used to:

[0236] Calling a text recognition model to perform text recognition on the at least one first text region image to obtain a prediction probability distribution corresponding to the first text region image, the prediction probability distribution including a character code prediction probability at each position in the first text region image;

[0237] Determining a third character code prediction sequence corresponding to the first text region image according to the character code prediction probabilities of the respective positions;

[0238] Determine a character decoding prediction sequence based on the third character encoding prediction sequence and the character decoding rule;

[0239] The target text in the original text image is determined based on the character decoding prediction sequence.

[0240] Optionally, the characters in the first text area image are characters corresponding to the target language, the target language includes multiple semantic characters, and each semantic character corresponds to multiple deformed characters; the character encoding rule includes a mapping relationship between text encoding and character encoding sequence, the character decoding rule includes a mapping relationship between character encoding sequence and character decoding sequence, the character encoding sequence includes the encoding of the semantic characters, and the character decoding sequence includes the encoding of the deformed characters.

[0241] In a specific implementation, the processor 601, the storage device 602, and the communication interface 603 described in the embodiment of the present application can execute the embodiment of the present application. Figure 2 or Figure 3 The implementation method described in the related embodiments of the provided text recognition method can also be used to implement the embodiments of this application. Figure 5 The implementation described in the relevant embodiments of the provided text recognition device will not be repeated here.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices, and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is merely a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical, or other forms.

[0243] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned text recognition device, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the above text recognition device. Figure 2 、 Figure 3 The method in the corresponding embodiment will therefore not be described in detail here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected by a communication network. Multiple computer devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.

[0244] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can perform the above-mentioned Figure 2 、 Figure 3 The method in the corresponding embodiment will therefore not be described in detail here.

[0245] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0246] The above disclosure is only part of the embodiments of the present application, and it is certainly not intended to limit the scope of the rights of the present application. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the invention.

Claims

1. A text recognition method, characterized in that: The method comprises: Get the original image; Segmenting the original image to obtain at least one first text area image; Calling a text recognition model to perform text recognition on the at least one first text area image to obtain a target text in the original image; the characters in the first text area image are characters corresponding to a target language, the target language includes multiple semantic characters, each semantic character corresponds to multiple deformed characters, and the target language is an agglutinative language; the text recognition model is obtained by training a first neural network and a second neural network included in an initial recognition model based on a text detection model, a sample image, and first supervisory information; the first supervisory information includes a character encoding reference sequence obtained by processing a reference text corresponding to the sample image based on a character encoding rule; the character encoding rule is used to convert the text encoding of the text into a first character encoding sequence, and to convert the first character encoding sequence into a second character encoding sequence; the first character encoding sequence includes encodings of semantic characters, and the second character encoding sequence includes encodings of deformed characters corresponding to the semantic characters, so as to distinguish different character deformation forms; the first neural network includes a continuous temporal classification network, and the second neural network includes a bidirectional long short-term memory network and an attention mechanism network; In which, the training method of the text detection model includes: obtaining a sample image and second supervisory information, the second supervisory information including one or more of the reference pixel type, reference connection type between pixels, and reference rotation angle of the third text area image in the sample image; calling the initial detection model to process the third text area image to obtain predicted feature data of the third text area image, the predicted feature data including one or more of the predicted pixel type, predicted connection type between pixels, and predicted rotation angle; adjusting the parameters of the initial detection model based on the second supervisory information and the predicted feature data to obtain a text detection model.

2. The method according to claim 1, characterized in that The method further comprises: Calling a text detection model to process the sample image to obtain at least one second text region image corresponding to the sample image; acquiring a character encoding reference sequence corresponding to the second text region image based on a reference text corresponding to the second text region image; calling a first neural network and a second neural network included in an initial recognition model to process the second text region image respectively to obtain a first character code prediction sequence and a second character code prediction sequence; Based on the character code reference sequence, the first character code prediction sequence and the second character code prediction sequence, parameters of the first neural network and the second neural network included in the initial recognition model are adjusted to obtain a text recognition model.

3. The method according to claim 2, characterized in that The step of adjusting parameters of a first neural network and a second neural network included in the initial recognition model based on the character code reference sequence, the first character code prediction sequence, and the second character code prediction sequence to obtain a text recognition model includes: determining a first difference between the character encoding reference sequence and the first character encoding prediction sequence, and a second difference between the character encoding reference sequence and the second character encoding prediction sequence; Adjusting network parameters of a first neural network and a second neural network included in the initial recognition model according to the first difference and the second difference; The first neural network after parameter adjustment is used as a text recognition model, and the character decoding rules are embedded in the text recognition model.

4. The method according to claim 2, characterized in that The obtaining, based on the reference text corresponding to the second text region image, a character encoding reference sequence corresponding to the second text region image includes: Obtaining a reference text corresponding to the second text region image; performing a first encoding process on a reference text corresponding to the second text region image to obtain a text code; A second encoding process is performed on the text encoding based on a character encoding rule to obtain a character encoding reference sequence corresponding to the second text region image.

5. The method according to claim 1, wherein The predicted feature data includes the predicted pixel type, the predicted connection type between the pixels, and the predicted rotation angle; and calling the initial detection model to process the third text region image to obtain the predicted feature data of the third text region image includes: calling an initial detection model to perform text detection on the third text region image to obtain a text region and a non-text region in the third text region image, and determining a predicted pixel type for each pixel according to a position of each pixel in the third text region image, the text region, and the non-text region; calling the initial detection model to process the predicted pixel types of two pixels included in each adjacent pixel pair in the third text region image to obtain a predicted connection type between corresponding pixels; The initial detection model is called to determine a predicted rotation angle of the characters in the third text region image.

6. The method according to any one of claims 1 to 5, characterized in that The text recognition model includes character decoding rules; calling the text recognition model to perform text recognition on the at least one first text region image to obtain the target text in the original image includes: Calling a text recognition model to perform text recognition on the at least one first text region image based on a character encoding rule to obtain a prediction probability distribution corresponding to the first text region image, the prediction probability distribution including a character encoding prediction probability for each position in the first text region image; determining a third character code prediction sequence corresponding to the first text region image according to the character code prediction probabilities of the respective positions; The target text in the original text image is determined according to the third character code prediction sequence and the character decoding rule.

7. A text recognition device, characterized in that: The device comprises: An acquisition module is used to acquire the original image; a processing module, configured to segment the original image to obtain at least one first text region image; A recognition module is configured to call a text recognition model to perform text recognition on the at least one first text region image to obtain a target text in the original image; the characters in the first text region image are characters corresponding to a target language, the target language includes multiple semantic characters, each semantic character corresponds to multiple deformed characters, and the target language is an agglutinative language; the text recognition model is obtained by training a first neural network and a second neural network included in an initial recognition model based on a text detection model, a sample image, and first supervisory information; the first supervisory information includes a character encoding reference sequence obtained by processing a reference text corresponding to the sample image based on a character encoding rule; the character encoding rule is used to convert a text encoding of a text into a first character encoding sequence, and to convert the first character encoding sequence into a second character encoding sequence; the first character encoding sequence includes encodings of semantic characters, and the second character encoding sequence includes encodings of deformed characters corresponding to the semantic characters, so as to distinguish different character deformation forms; the first neural network includes a continuous temporal classification network, and the second neural network includes a bidirectional long short-term memory network and an attention mechanism network; In which, the training method of the text detection model includes: obtaining a sample image and second supervisory information, the second supervisory information including one or more of the reference pixel type, reference connection type between pixels, and reference rotation angle of the third text area image in the sample image; calling the initial detection model to process the third text area image to obtain predicted feature data of the third text area image, the predicted feature data including one or more of the predicted pixel type, predicted connection type between pixels, and predicted rotation angle; adjusting the parameters of the initial detection model based on the second supervisory information and the predicted feature data to obtain a text detection model.

8. A computer device, characterized in that: include: A processor, a storage device, and a communication interface, wherein the processor, the communication interface, and the storage device are interconnected, wherein the storage device stores an executable program code, and the processor is used to call the executable program code to implement the text recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions are executed by a processor to implement the text recognition method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, it is used to implement the text recognition method according to any one of claims 1 to 6.

Images