Image text language recognition method and system

Abstract

A method and system for image text language recognition, the method simulates image text in real scenes, combines background pictures, dictionary libraries, thesaurus libraries and font libraries of various languages, artificially synthesizes annotated image text to form a first training data set; collects image text in real scenes, and manually classifies and annotates the collected image text to form a second training data set; constructs a neural network CRNN for OCR language recognition, uses the first training data set for preliminary training, and obtains an OCR text recognition model; uses the first training data set and the second training data set to fine-tun the OCR text recognition model to obtain an OCR language recognition classification model; and uses the fine-tuned OCR language recognition classification model for model reasoning. The present invention can obtain a better fitting effect, reduce missed detection or false positives, and improve the performance of the model.

Description

Technical Field

[0001] The present invention relates to the field of OCR text processing technology, and in particular to an image text language recognition method and system. Background Art

[0002] At present, in Internet scenarios, especially in scenarios such as overseas cooperation, cross-border trade, and international advertising, the same picture often contains text in multiple languages ​​at the same time. The language selection based on a priori is not enough to cover all scenario requirements. Therefore, language recognition is necessary before text recognition.

[0003] In the prior art, multilingual OCR language recognition solutions usually include a general image recognition model and a multi-label-based image recognition model, which directly extracts the semantic features of the entire sentence of image text and then classifies it through a feature classifier. Traditional technology only considers the semantic features at the image level, and does not consider the relationship between contexts based on the sequence, and is unable to distinguish between languages ​​with similar shape features at the image level. At the same time, when more than two languages ​​appear in the entire sentence of the image text, and one of the languages ​​accounts for more than 80%, the multi-label-based image recognition method often misses detection or makes mistakes. In summary, there is an urgent need for a new technical solution for image text language recognition. Summary of the invention

[0004] To this end, the present invention provides an image text language recognition method and system to solve the problems of missed detection or false detection and poor performance in traditional solutions.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for identifying the language of an image text, comprising:

[0006] Simulate the image text in the real scene, combine the background picture, dictionary library, thesaurus library and font library of various languages ​​in the simulation process, and artificially synthesize the annotated image text to form the first training data set;

[0007] Collecting images and texts in real scenes, and manually classifying and labeling the collected images and texts to form a second training data set;

[0008] Constructing a neural network CRNN for OCR language recognition, and using the first training data set to perform preliminary training on the constructed neural network CRNN to obtain an OCR text recognition model;

[0009] Fine-tuning the OCR text recognition model using the first training data set and the second training data set to obtain an OCR language recognition classification model;

[0010] The fine-tuned OCR language recognition and classification model is used for model reasoning.

[0011] As a preferred solution for image text language recognition, the simulation process of image text in real scenes is carried out:

[0012] Collect a dataset of images without text from the Internet and crop them to serve as background images for artificially synthesized images and text;

[0013] Dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and font libraries of corresponding languages ​​are obtained from official websites of various languages ​​to synthesize image texts of different styles.

[0014] As a preferred solution of the image text language recognition method, the dictionary library and the lexicon library are numbered so that all characters correspond to unique indexes, and are saved in a specified format as labels of artificially synthesized image texts;

[0015] Each training sample includes language-level annotation information and text-level annotation information; the text-level annotation information formed according to the unique index is used for the OCR text recognition model; the language-level annotation information is used for fine-tuning the language recognition classification model.

[0016] As a preferred solution of the image text language recognition method, the neural network CRNN includes a convolutional neural network CNN and a recurrent neural network RNN;

[0017] The convolutional neural network CNN uses the backbone part of SE_ResNeXt50_32x4d to extract image semantic features, and 2 layers of Bi-directional LSTM are connected after SE_ResNeXt50_32x4d to extract image text sequence features; based on CTC_loss, the first training data set is used to train the OCR text recognition model.

[0018] As a preferred solution for the image text language recognition method, during the fine-tuning process of the OCR text recognition model, the first training data set and the second training data set are mixed into one training data set according to a preset distribution ratio, language-level annotation information is used, the backbone part and Bi-directional LSTM layer of SE_ResNeXt50_32x4d are frozen, and the text recognition classifier is replaced with a language recognition classifier for fine-tuning.

[0019] The present invention also provides an image text language recognition system, comprising:

[0020] The first data set generation module is used to simulate image text in real scenes. The simulation process combines background pictures, dictionary libraries, lexicon libraries and font libraries of various languages ​​to artificially synthesize annotated image text to form a first training data set;

[0021] A second data set generation module is used to collect images and texts in real scenes, and manually classify and annotate the collected images and texts to form a second training data set;

[0022] A text recognition processing module, used to construct a neural network CRNN for OCR language recognition, and to perform preliminary training on the constructed neural network CRNN using the first training data set to obtain an OCR text recognition model;

[0023] A language recognition and classification module, configured to fine-tune the OCR text recognition model using the first training data set and the second training data set to obtain an OCR language recognition and classification model;

[0024] The model inference module is used to perform model inference using the fine-tuned OCR language recognition and classification model.

[0025] As a preferred solution of the image text language recognition system, in the first data set generation module, a simulation process is performed on the image text in the real scene:

[0026] Collect a dataset of images without text from the Internet and crop them to serve as background images for artificially synthesized images and text;

[0027] Dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and font libraries of corresponding languages ​​are obtained from official websites of various languages ​​to synthesize image texts of different styles.

[0028] As a preferred solution of the image text language recognition system, in the first data set generation module, the dictionary library and the lexicon library are numbered so that all characters correspond to unique indexes and are saved in a specified format as labels of artificially synthesized image texts;

[0029] In the second data set generation module, each training sample includes language-level annotation information and text-level annotation information; the text-level annotation information formed according to the unique index is used for the OCR text recognition model; the language-level annotation information is used for fine-tuning the language recognition classification model.

[0030] As a preferred solution of the image text language recognition system, the neural network CRNN in the text recognition processing module includes a convolutional neural network CNN and a recurrent neural network RNN;

[0031] The convolutional neural network CNN uses the backbone part of SE_ResNeXt50_32x4d to extract image semantic features, and 2 layers of Bi-directional LSTM are connected after SE_ResNeXt50_32x4d to extract image text sequence features; based on CTC_loss, the first training data set is used to train the OCR text recognition model.

[0032] As a preferred solution for the image text language recognition system, during the fine-tuning process of the OCR text recognition model, the language recognition classification module mixes the first training data set and the second training data set into one training data set according to a preset distribution ratio, uses language-level annotation information, freezes the backbone part and Bi-directional LSTM layer of SE_ResNeXt50_32x4d, and replaces the text recognition classifier with a language recognition classifier for fine-tuning.

[0033] The present invention has the following advantages: by simulating image text in real scenes, the simulation process combines background pictures, dictionary libraries, thesaurus libraries and font libraries of various languages, and artificially synthesizes annotated image text to form a first training data set; collects image text in real scenes, and manually classifies and annotates the collected image text to form a second training data set; constructs a neural network CRNN for OCR language recognition, and uses the first training data set to perform preliminary training on the constructed neural network CRNN to obtain an OCR text recognition model; uses the first training data set and the second training data set to fine-tune the OCR text recognition model to obtain an OCR language recognition classification model; and uses the fine-tuned OCR language recognition classification model to perform model reasoning. The present invention can obtain a better fitting effect, reduce missed detection or false positives, and improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the drawings required for the implementation methods or the prior art descriptions are briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0035] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0036] Figure 1 A schematic diagram of a flow chart of a method for identifying the language of an image and text provided in Embodiment 1 of the present invention;

[0037] Figure 2 A flow chart of synthesizing text images of multiple languages ​​and different styles in the method for identifying the language of text in an image provided by Embodiment 1 of the present invention;

[0038] Figure 3 The format for storing the artificially synthesized text image label in the image text language recognition method provided in Example 1 of the present invention;

[0039] Figure 4 A detailed diagram of the SE_ResNeXt50_32x4d model block in the image text language recognition method provided in Example 1 of the present invention;

[0040] Figure 5 The overall framework of CRNN and modification details of SE_ResNeXt50_32x4d in the image text language recognition method provided in Example 1 of the present invention;

[0041] Figure 6 The CRNN model inference configuration in the image text language recognition method provided in Example 1 of the present invention;

[0042] Figure 7 A schematic diagram of an image text language recognition system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0043] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Example 1

[0045] See also Figure 1 , Figure 2 and Figure 3Embodiment 1 of the present invention provides a method for identifying the language of an image text, comprising:

[0046] S1, simulating image text in real scenes, combining background images, dictionary libraries, thesaurus libraries and font libraries of various languages, and artificially synthesizing annotated image text to form a first training data set;

[0047] S2, collecting images and texts in real scenes, and manually classifying and labeling the collected images and texts to form a second training data set;

[0048] S3, constructing a neural network CRNN for OCR language recognition, and using the first training data set to perform preliminary training on the constructed neural network CRNN to obtain an OCR text recognition model;

[0049] S4, fine-tuning the OCR text recognition model using the first training data set and the second training data set to obtain an OCR language recognition classification model;

[0050] S5. Perform model reasoning using the fine-tuned OCR language recognition and classification model.

[0051] In this embodiment, step S1 is to simulate the image text in the real scene:

[0052] Collect a dataset of images without text from the Internet and crop them to serve as background images for artificially synthesized images and text;

[0053] Dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and font libraries of corresponding languages ​​are obtained from official websites of various languages ​​to synthesize image texts of different styles.

[0054] Specifically, in the process of simulating image text in real scenes and artificially synthesizing training data to form the first training data set, firstly, a picture data set without text is collected from the Internet, and the 32*352 in this example is cropped as the background picture of the artificially synthesized image text; then, dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and this example includes: Latin, simplified Chinese, traditional Chinese, Japanese, Korean, Devanagari, Tamil, Thai and Arabic; among them, Chinese, Japanese and Korean generate dictionary libraries in units of characters, and Latin, Devanagari, Tamil, Thai and Arabic generate dictionary libraries in units of words. In addition, it is necessary to pay attention to the writing order of Arabic from right to left; secondly, the font library (font file) of the corresponding language is obtained from the official website of each language or other channels for synthesizing image text of different styles; in this way, based on the above-mentioned picture text background, dictionary library and lexicon library, as well as font files of font libraries of various language styles, the image text is synthesized.

[0055] In this embodiment, the dictionary library and the lexicon library are numbered so that all characters correspond to unique indexes and are saved in a specified format as labels of artificially synthesized image texts;

[0056] Each training sample includes language-level annotation information and text-level annotation information; the text-level annotation information formed according to the unique index is used for the OCR text recognition model; the language-level annotation information is used for fine-tuning the language recognition classification model.

[0057] Assistance Figure 3 Specifically, the dictionary library (the dictionary library needs to be deduplicated) and the lexicon library are numbered from 0 to ensure that all characters correspond to unique indexes, and are saved in json format as labels for all artificially synthesized image texts, indicating that each training sample contains not only language-level annotation information, but also each text-level annotation information in the image, where the text-level annotation information formed according to the unique index is used for the OCR text recognition model in step S3, and the language-level annotation information is used to fine-tune the OCR language recognition classification model in step S4.

[0058] In this embodiment, the neural network CRNN includes a convolutional neural network CNN and a recurrent neural network RNN;

[0059] The convolutional neural network CNN uses the backbone part of SE_ResNeXt50_32x4d to extract image semantic features, and 2 layers of Bi-directional LSTM are connected after SE_ResNeXt50_32x4d to extract image text sequence features; based on CTC_loss, the first training data set is used to train the OCR text recognition model.

[0060] Specifically, the CRNN neural network constructed in step S3 includes two parts: a convolutional neural network (CNN) and a recurrent neural network (RNN). The CNN structure uses the backbone of SE_ResNeXt50_32x4d for image semantic feature extraction, followed by a 2-layer Bi-directional LSTM for extracting image text sequence features. Finally, based on CTC_loss, a multilingual text recognition model is trained using an artificially synthesized training data set.

[0061] This example modifies the SE_ResNeXt50_32x4d (cardinality=32,dimensionreduction=4) network of the CNN backbone to ensure that the resolution of layer2, layer3 and layer4 in the X-axis direction remains unchanged when they are downsampled for the first time, and the Y-axis remains downsampled by 2 times, that is, stride=(1,2). The final input is (32*352*channel), the backbone output is (1*88*channel), and the extracted feature map becomes 1 / 4 of the original image in the X-axis direction and 1 / 32 of the original image in the Y-axis direction. Figure 4 and Figure 5 In this example, CTC_loss is used to solve the problem of mismatch between model output and label length. The implementation formula is as follows:

[0062] Given an input x, the probability that the output of LSTM is l is:

[0063]

[0064] where π∈B -1 (l) represents all paths π that are l after transformation by B.

[0065] Among them, for any path formula is:

[0066] In this embodiment, during the fine-tuning process of the OCR text recognition model, the first training data set and the second training data set are mixed into one training data set according to a preset distribution ratio, language-level annotation information is used, the backbone part and the Bi-directional LSTM layer of SE_ResNeXt50_32x4d are frozen, and the text recognition classifier is replaced with a language recognition classifier for fine-tuning.

[0067] Specifically, the artificially synthesized image text training data and the real scene image text training data are mixed into a training data set in a 1:1 distribution ratio. The labels used are all language-level labels. The SE_ResNeXt50_32x4d backbone part and LSTM layer of the model are frozen, and the multilingual text recognition classifier is replaced with a language recognition classifier for fine-tuning.

[0068] See also Figure 6 In order to ensure batch processing during the inference process, the input image size is fixed at 32*352. If the input image width is less than 352, the original image is copied until it is filled to 352. If the width is greater than 352, the center of the image is used as the origin, and sub-images with a width of 352 are cropped on both sides for prediction.

[0069] In summary, the present invention simulates image text in real scenes, and the simulation process combines background images, dictionary libraries, thesaurus libraries and font libraries of various languages ​​to artificially synthesize annotated image text to form a first training data set; collects image text in real scenes, and manually classifies and annotates the collected image text to form a second training data set; constructs a neural network CRNN for OCR language recognition, and uses the first training data set to perform preliminary training on the constructed neural network CRNN to obtain an OCR text recognition model; uses the first training data set and the second training data set to fine-tune the OCR text recognition model to obtain an OCR language recognition classification model; and uses the fine-tuned OCR language recognition classification model to perform model reasoning. The present invention can obtain a better fitting effect, reduce missed detection or false positives, and improve the performance of the model.

[0070] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0071] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0072] Example 2

[0073] See also Figure 7 Embodiment 2 of the present invention provides an image text language recognition system, comprising:

[0074] The first data set generation module 1 is used to simulate image text in real scenes. The simulation process combines background pictures, dictionary libraries, lexicon libraries and font libraries of various languages ​​to artificially synthesize annotated image text to form a first training data set;

[0075] The second data set generation module 2 is used to collect images and texts in real scenes, and manually classify and annotate the collected images and texts to form a second training data set;

[0076] The text recognition processing module 3 is used to construct a neural network CRNN for OCR language recognition, and use the first training data set to perform preliminary training on the constructed neural network CRNN to obtain an OCR text recognition model;

[0077] A language recognition and classification module 4 is used to fine-tune the OCR text recognition model using the first training data set and the second training data set to obtain an OCR language recognition and classification model;

[0078] The model inference module 5 is used to perform model inference using the fine-tuned OCR language recognition and classification model.

[0079] In this embodiment, in the first data set generation module 1, a simulation process is performed on image text in a real scene:

[0080] Collect a dataset of images without text from the Internet and crop them to serve as background images for artificially synthesized images and text;

[0081] Dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and font libraries of corresponding languages ​​are obtained from official websites of various languages ​​to synthesize image texts of different styles.

[0082] In this embodiment, in the first data set generating module 1, the dictionary library and the lexicon library are numbered so that all characters correspond to unique indexes and are saved in a specified format as labels of artificially synthesized image texts;

[0083] In the second data set generation module 2, each training sample includes language-level annotation information and text-level annotation information; the text-level annotation information formed according to the unique index is used for the OCR text recognition model; the language-level annotation information is used for fine-tuning the language recognition classification model.

[0084] In this embodiment, the neural network CRNN in the text recognition processing module 3 includes a convolutional neural network CNN and a recurrent neural network RNN;

[0085] The convolutional neural network CNN uses the backbone part of SE_ResNeXt50_32x4d to extract image semantic features, and 2 layers of Bi-directional LSTM are connected after SE_ResNeXt50_32x4d to extract image text sequence features; based on CTC_loss, the first training data set is used to train the OCR text recognition model.

[0086] In this embodiment, during the fine-tuning process of the OCR text recognition model, the language recognition classification module 4 mixes the first training data set and the second training data set into one training data set according to a preset distribution ratio, uses language-level annotation information, freezes the backbone part and the Bi-directionalLSTM layer of SE_ResNeXt50_32x4d, and replaces the text recognition classifier with a language recognition classifier for fine-tuning.

[0087] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous part of the present application, and will not be repeated here.

[0088] Example 3

[0089] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code of an image text language recognition method is stored. The program code includes instructions for executing the image text language recognition method of embodiment 1 or any possible implementation thereof.

[0090] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0091] Example 4

[0092] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0093] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the image text language recognition method of embodiment 1 or any possible implementation method thereof.

[0094] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0095] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0096] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0097] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for identifying the language of an image, characterized in that: include: Simulate the image text in the real scene, combine the background picture, dictionary library, thesaurus library and font library of various languages ​​in the simulation process, and artificially synthesize the annotated image text to form the first training data set; Collecting images and texts in real scenes, and manually classifying and labeling the collected images and texts to form a second training data set; Constructing a neural network CRNN for OCR language recognition, and using the first training data set to perform preliminary training on the constructed neural network CRNN to obtain an OCR text recognition model; Fine-tuning the OCR text recognition model using the first training data set and the second training data set to obtain an OCR language recognition classification model; Perform model reasoning using the fine-tuned OCR language recognition and classification model; The dictionary library and the lexicon library are numbered so that all characters correspond to unique indexes, and are saved in a specified format as labels of artificially synthesized image texts; Each training sample includes language-level annotation information and text-level annotation information; the text-level annotation information formed according to the unique index is used for the OCR text recognition model; the language-level annotation information is used for fine-tuning the language recognition classification model; The neural network CRNN includes a convolutional neural network CNN and a recurrent neural network RNN; The convolutional neural network CNN uses the backbone of SE_ResNeXt50_32x4d to extract image semantic features, and a 2-layer Bi-directional LSTM is connected after SE_ResNeXt50_32x4d to extract image text sequence features; Based on CTC_loss, the first training data set is used to train the OCR text recognition model; In the process of fine-tuning the OCR text recognition model, the first training data set and the second training data set are mixed into one training data set according to a preset distribution ratio, the language-level annotation information is used, the backbone part and the Bi-directional LSTM layer of SE_ResNeXt50_32x4d are frozen, and the text recognition classifier is replaced with the language recognition classifier for fine-tuning; The SE_ResNeXt50_32x4d network of the CNN backbone is modified to ensure that the resolution of layer2, layer3 and layer4 in the X-axis direction remains unchanged when they are downsampled for the first time, and the Y-axis is kept downsampled by 2 times, that is, stride = (1,2). The final input is (32*352*channel), the backbone output is (1*88*channel), and the extracted feature map becomes 1 / 4 of the original image in the X-axis direction and 1 / 32 of the original image in the Y-axis direction. The CTC_loss implementation formula is as follows: Given an input x, the probability that the output of LSTM is l is: where π∈B -1 (l) represents all paths π that are l after transformation by B.

2. The method for identifying the language of an image text according to claim 1, characterized in that: Simulating the image text in real scenes: Collect a dataset of images without text from the Internet and crop them to serve as background images for artificially synthesized images and text; Dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and font libraries of corresponding languages ​​are obtained from official websites of various languages ​​to synthesize image texts of different styles.

3. An image text language recognition system, characterized in that: include: The first data set generation module is used to simulate image text in real scenes. The simulation process combines background pictures, dictionary libraries, lexicon libraries and font libraries of various languages ​​to artificially synthesize annotated image text to form a first training data set; A second data set generation module is used to collect images and texts in real scenes, and manually classify and annotate the collected images and texts to form a second training data set; A text recognition processing module, used to construct a neural network CRNN for OCR language recognition, and to perform preliminary training on the constructed neural network CRNN using the first training data set to obtain an OCR text recognition model; A language recognition and classification module, configured to fine-tune the OCR text recognition model using the first training data set and the second training data set to obtain an OCR language recognition and classification model; A model inference module, used for performing model inference using the fine-tuned OCR language recognition and classification model; In the first data set generation module, the dictionary library and the lexicon library are numbered so that all characters correspond to unique indexes and are saved in a specified format as labels of artificially synthesized image texts; In the second data set generation module, each training sample includes language-level annotation information and text-level annotation information; the text-level annotation information formed according to the unique index is used for the OCR text recognition model; the language-level annotation information is used for fine-tuning the language recognition classification model; The neural network CRNN in the text recognition processing module includes a convolutional neural network CNN and a recurrent neural network RNN; The convolutional neural network CNN uses the backbone of SE_ResNeXt50_32x4d to extract image semantic features, and a 2-layer Bi-directional LSTM is connected after SE_ResNeXt50_32x4d to extract image text sequence features; Based on CTC_loss, the first training data set is used to train the OCR text recognition model; During the fine-tuning process of the OCR text recognition model by the language recognition classification module, the first training data set and the second training data set are mixed into one training data set according to a preset distribution ratio, the language-level annotation information is used, the backbone part and the Bi-directional LSTM layer of SE_ResNeXt50_32x4d are frozen, and the text recognition classifier is replaced with the language recognition classifier for fine-tuning; The SE_ResNeXt50_32x4d network of the CNN backbone is modified to ensure that the resolution of layer2, layer3 and layer4 in the X-axis direction remains unchanged when they are downsampled for the first time, and the Y-axis is kept downsampled by 2 times, that is, stride = (1,2). The final input is (32*352*channel), the backbone output is (1*88*channel), and the extracted feature map becomes 1 / 4 of the original image in the X-axis direction and 1 / 32 of the original image in the Y-axis direction. The CTC_loss implementation formula is as follows: Given an input x, the probability that the output of LSTM is l is: where π∈B -1 (l) represents all paths π that are l after transformation by B.

4. The image text language recognition system according to claim 3, characterized in that: In the first data set generation module, the image text in the real scene is simulated: Collect a dataset of images without text from the Internet and crop them to serve as background images for artificially synthesized images and text; Dictionary libraries or lexicon libraries of various languages ​​are collected from the Internet, and font libraries of corresponding languages ​​are obtained from official websites of various languages ​​to synthesize image texts of different styles.

Images