A method and device for recognizing a multilingual text, an electronic device and a medium

Abstract

The application provides a multilingual text recognition method, device and equipment and medium, the method comprises: obtaining an image to be recognized containing multilingual text; using a Transformer encoder to extract features of the image to be recognized to obtain image features; based on the image features, determining the text position of the image to be recognized through a position judgment neural network, and determining the language of the text through a language judgment neural network; according to the text position and the language of the text, the image to be recognized is divided into multiple local images, each local image contains single language text; after preprocessing the divided local image, the CTC technology is used for decoding to obtain the final text recognition result. The performance and accuracy of multilingual text recognition are improved.

Description

Technical Field

[0001] This disclosure relates to the technical field of text recognition, and in particular to a method, apparatus, electronic device, and medium for recognizing multilingual text. Background Technology

[0002] With accelerating globalization, the demand for technologies capable of processing and understanding multilingual text is increasing. Multilingual text recognition technology has wide applications in various fields, including international communication, information retrieval, machine translation, and cross-cultural communication. Traditional optical character recognition (OCR) technology is primarily designed and optimized for single-language text, but it faces challenges when processing text containing multiple languages. These challenges mainly include the differences in character sets between different languages, the specificities of language structures, and the problem of automatic language identification in mixed text contexts.

[0003] In recent years, the rise of deep learning technology has brought new breakthroughs to multilingual text recognition. In particular, the introduction of the Transformer encoder, with its self-attention mechanism, has successfully captured long-distance dependencies in sequential data, significantly improving the model's ability to learn different linguistic features. Furthermore, combining position-determining neural networks and language-determining neural networks can effectively achieve precise text location and accurate language identification in complex multilingual environments. However, despite the progress made by existing multilingual OCR systems, they still have shortcomings in the accuracy of recognizing mixed multilingual text. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this disclosure is to provide a method, apparatus, electronic device and medium for recognizing multilingual text, with the aim of improving the performance and accuracy of multilingual text recognition.

[0005] The first aspect of this disclosure provides a method for recognizing multilingual text, comprising: acquiring an image to be recognized containing text in multiple languages; extracting features from the image to be recognized using a Transformer encoder to obtain image features; determining the text position of the image to be recognized using a position determination neural network and determining the text language using a language determination neural network based on the image features; segmenting the image to be recognized into multiple local images according to the text position and the text language, each local image containing text in a single language; and preprocessing the segmented local images and then decoding them using CTC technology to obtain the final text recognition result.

[0006] In an embodiment of the first aspect, the method further includes: the input of the location determination neural network is a location subset of the image features, the location subset being used for determining text location; the output of the location determination neural network is the probability that text exists at each pixel location; when the probability that text exists is greater than a threshold, the corresponding pixel location is determined as the text location.

[0007] In an embodiment of the first aspect, the method further includes: the input of the language determination neural network is a language seed set of the image features, the language seed set being used for determining the language of the text; the output of the language determination neural network is a text language probability distribution; the language with the highest probability in the language probability distribution is determined as the language of the corresponding pixel position.

[0008] In the first aspect of the embodiment, the location determination neural network employs a multilayer perceptron network, and its output layer uses the sigmoid activation function; the language determination neural network employs a multilayer perceptron network, and its output layer uses the softmax activation function.

[0009] In an embodiment of the first aspect, after segmenting the image to be identified into multiple local images, the method further includes: embedding special characters representing a specific language before and after each local image to indicate the text language of each local image.

[0010] In the first aspect of the embodiment, the segmented local images are preprocessed and then decoded using CTC technology to obtain the final text recognition result. This includes: extracting special character features of each local image using a multilayer perceptron network and obtaining visual features of each local image using a visual model; concatenating the special character features and the visual features of the local images to form a new input matrix; encoding the new input matrix using a neural network language model to generate a feature matrix; and inputting the feature matrix into a CTC decoder for decoding to obtain the final text recognition result.

[0011] In the first aspect of the embodiment, after obtaining the final text recognition result, the method further includes: matching the decoded text with a preset special character library to determine and separate the text content and corresponding language information of each local image; and converting the text content into a unified representation format or translating it into the target language based on the separated language information for subsequent use.

[0012] A second aspect of this disclosure discloses a multilingual text recognition device, comprising: an acquisition module for acquiring an image to be recognized containing text in multiple languages; a feature extraction module for extracting features from the image to be recognized using a Transformer encoder to obtain image features; a judgment module for determining the text position of the image to be recognized based on the image features using a position judgment neural network and determining the text language using a language judgment neural network; a segmentation module for segmenting the image to be recognized into multiple local images according to the text position and the text language, each local image containing text in a single language; and a recognition module for preprocessing the segmented local images and then decoding them using CTC technology to obtain the final text recognition result.

[0013] A third aspect of this disclosure discloses an electronic device comprising: a processor and a memory; wherein the memory is configured to store a computer program; and the processor is configured to execute the computer program stored in the memory to cause the electronic device to perform a multilingual text recognition method as described in any of the first aspects.

[0014] The fourth aspect of this disclosure discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the multilingual text recognition method described in any of the first aspects.

[0015] As described above, the multilingual text recognition method, apparatus, electronic device, and medium disclosed herein, by using a Transformer encoder to extract features from images and combining a position-determining neural network and a language-determining neural network, can more accurately recognize multilingual text in images. Simultaneously, it can handle text of different sizes, shapes, and backgrounds, thereby improving the robustness of recognition. Furthermore, by segmenting the image to be recognized into multiple local images, each containing text in a single language, the text content in each local image can be more accurately identified, improving recognition accuracy. Further, using CTC technology to decode the local images allows for rapid acquisition of the final text recognition result, improving recognition speed and efficiency. Additionally, embedding special characters representing specific languages ​​before and after each local image can more accurately indicate the language of the text in each local image, aiding subsequent text recognition and processing. Attached Figure Description

[0016] Figure 1 The illustration shows a scenario of an electronic device of this disclosure in one embodiment;

[0017] Figure 2 A flowchart illustrating a method for recognizing multilingual text in one embodiment of this disclosure is shown.

[0018] Figure 3This illustration shows a flowchart of local image preprocessing in one embodiment of the present disclosure.

[0019] Figure 4 This disclosure demonstrates the complete processing steps of a specific example.

[0020] Figure 5 This illustration shows a flowchart of the post-processing of text recognition results in one embodiment of the present disclosure.

[0021] Figure 6 A schematic diagram of a multilingual text recognition device is shown in one embodiment of this disclosure.

[0022] Figure 7 A schematic diagram of the circuit structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] The following embodiments of the present invention provide a method for recognizing multilingual text, which can be applied to, for example... Figure 1 The electronic devices shown are as follows. The electronic devices described in this invention may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This invention does not impose any restrictions on the specific type of electronic device.

[0026] For example, the electronic device may be a station (STAION, ST) in a WLAN with wireless charging capability, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, Personal Digital Assistant (PDA) device, handheld device with wireless charging capability, computing device or other processing device, computer, laptop computer, handheld communication device, handheld computing device, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0027] For example, the electronic device can communicate with networks and other devices wirelessly. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0028] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] like Figure 2 The diagram shows a flowchart of a multilingual text recognition method according to an embodiment of the first aspect of this disclosure, including steps S21-S25.

[0030] Step S21: Obtain the image to be recognized, which contains text in multiple languages.

[0031] Specifically, the image to be recognized can be captured by a digital camera, smartphone camera, or other types of imaging devices. These devices capture images of real-world scenes in various environments, such as street signs, book pages, or documents, which may contain text in one or more languages. Alternatively, for paper documents, digitization using a scanner is a common option. Scanners can capture images at high resolution, thus preserving details in the document for easier subsequent processing. In some systems, users can directly upload image files already stored on a computer or mobile device. Supported formats may include common image formats such as JPEG, PNG, and BMP. Especially in mobile applications, OCR systems may offer the ability to instantly capture and process images using the device's camera.

[0032] To improve recognition accuracy, image enhancement processing is necessary, such as adjusting contrast and brightness, and performing denoising operations to reduce random noise in the image. If the image is color, converting it to grayscale can reduce processing complexity, as color information is usually not necessary for text recognition. Converting the image to a binary image with only black and white pixels can help better distinguish text from the background. Additionally, to ensure the image meets the requirements of OCR processing, the image resolution may need to be adjusted. If the image is tilted, orientation correction is required to avoid affecting text recognition.

[0033] Step S22: Use a Transformer encoder to extract features from the image to be identified to obtain image features.

[0034] Specifically, the Transformer encoder is a deep learning model based on self-attention mechanisms, used to handle sequence-to-sequence problems. It consists of multiple identical layers, each comprising two main parts: a multi-head self-attention layer and a position-wise feedforward neural network. After each sub-layer, residual connections and layer normalization operations are used to help the model capture the dependencies at all positions in the input sequence.

[0035] First, the input image to be recognized is segmented into a series of small image patches. These patches serve as the input sequence for the Transformer encoder. Each patch consists of a set of pixels, which are converted into a series of numerical representations in subsequent steps for processing. Next, the pixel values ​​in each patch are converted into a one-dimensional vector. This is typically done by flattening and concatenating the pixel values ​​of the patch. To fit the Transformer model, these vectors are mapped to a fixed-size dimensional space through an embedding layer. Since the Transformer encoder itself cannot process data order, positional encodings are added to the vector of each image patch. This preserves the original order information even if the data is shuffled during the self-attention mechanism. Positional encoding is typically implemented by applying different sine and cosine functions to the vector at each position.

[0036] Then, a self-attention mechanism is used to process these image patch vectors enhanced with location information. Self-attention allows the model to capture global contextual information in the image by considering information from all other image patches when processing the current patch. This process is achieved by computing the query, key, and value, where Q, K, and V are all derived from the input image patch vectors. After self-attention, each image patch vector is passed through a feedforward neural network (FFN), which consists of two linear transformations with a ReLU activation function in between. The feedforward neural network helps the model learn more complex feature representations. After the FFN, layer normalization is applied to stabilize the training process and accelerate convergence. After self-attention and the FFN, residual connections are used to combine the input and output, which helps mitigate the information loss problem in deep networks. Transformer encoders typically consist of multiple such stacked layers, each extracting higher-level features until a rich image feature representation is ultimately formed.

[0037] Step S23: Based on the image features, determine the text position of the image to be identified through a position determination neural network, and determine the language of the text through a language determination neural network.

[0038] In some embodiments, the primary task of the location-determining neural network is to determine the specific location of text in an image. It receives image features from a Transformer encoder as input, which already contain visual information about the image. The location-determining neural network analyzes these features and outputs a probability map indicating the likelihood that each pixel or region contains text. This is typically implemented using a multilayer perceptron (MLP), whose output layer employs a sigmoid activation function to generate a probability value between 0 and 1 for each pixel, indicating whether text exists at that location. If the probability value at a location exceeds a preset threshold, then that location is identified as a text location.

[0039] It's important to note that the Sigmoid activation function is a non-linear function that compresses the input to a range between 0 and 1. It's commonly used in the output layer of neural networks to represent probabilities or perform binary classification. The Sigmoid function is defined as σ(x) = 1 / (1 + e^-x), where e is the base of the natural logarithm and x is the input value. The derivative of the Sigmoid function can be expressed as σ'(x) = σ(x)(1 - σ(x)). This property makes gradient calculation during backpropagation simple and efficient. The Sigmoid function maps any given input to the range of 0 to 1, making it well-suited for representing probabilities. Due to its sigmoid curve characteristic, the Sigmoid function provides the necessary non-linearity for neural networks, enabling them to learn and simulate complex data patterns. Another advantage of the Sigmoid function is the ease of calculating its derivative, which is very useful when optimizing gradient descent in neural networks. Because the output is constrained between 0 and 1, the Sigmoid function effectively normalizes all inputs.

[0040] Furthermore, after determining the location of the text, the next step is to identify its language. The language-determination neural network also receives image features from the Transformer encoder, particularly those related to the text location. The goal of this network is to output a language probability distribution, indicating the probability that the input text belongs to each of the different languages. This is typically achieved using another multilayer perceptron, whose output layer uses a softmax activation function to ensure that the sum of the probabilities for each language is 1. The language with the highest probability is determined as the language of the text.

[0041] It's important to note that the Softmax activation function is a computational method used to map a K-dimensional vector (real-valued) containing arbitrary real numbers to another K-dimensional real-valued vector, where each element takes a value between (0, 1). The Softmax function is commonly used in the output layer of multi-class classification problems, extending logistic regression, the sigmoid function, and other methods to multi-class problems. Given a K-dimensional real-valued vector Z, the Softmax function transforms it into a K-dimensional real-valued vector σ(Z), where each element σ(Z)_i is between 0 and 1, and the sum of all elements is 1. After processing with the Softmax function, the model's output can be interpreted as a probability distribution; that is, for a specific input sample, the model outputs a probability distribution representing the probability that the sample belongs to each class. In classification tasks, the class with the highest probability is usually selected as the final prediction. When training a classification model, the cross-entropy loss function is typically used to measure the difference between the model's predicted probability distribution and the true label. The combination of the Softmax function and the cross-entropy loss function not only gives the model output a clear probabilistic meaning but also facilitates the implementation of optimization algorithms such as gradient descent, because the gradient calculation of the Softmax function is relatively simple, which is beneficial for updating parameters during model training. The Softmax activation function is widely used in multi-class classification problems in deep learning and machine learning, such as image recognition, text classification, and speech recognition. In these applications, the Softmax layer is usually the last layer of the neural network, responsible for outputting the predicted probabilities of each class. Compared with activation functions such as Sigmoid and Tanh, the Softmax function is particularly suitable for solving multi-class classification problems where classes are mutually exclusive. Activation functions like ReLU and its variant Leaky ReLU are more often used in hidden layers to increase the model's non-linear expressive power.

[0042] Step S24: Based on the text location and the text language, the image to be identified is divided into multiple local images, each of which contains text in a single language.

[0043] Specifically, firstly, the image segmentation method needs to be determined based on the text location information provided by the location-based neural network. If the text lines are scattered, each line may need to be segmented individually; if the text is compact or close together, a more refined segmentation strategy may be required. After determining the segmentation strategy, the next step is to accurately segment the boundaries of each local image. This can be achieved through edge detection, connected component analysis, and other methods. For example, the Sobel or Canny operators in the OpenCV library can be used to detect text edges, thereby determining character boundaries. During segmentation, the spacing between characters needs to be considered to avoid segmenting adjacent characters into the same local image. Projection methods can be used to detect the spacing between characters, thus determining the segmentation points. Simultaneously, morphological operations such as erosion and dilation can be performed to help adjust the shape of the characters, making the segmentation more accurate. These operations can remove noise while maintaining the integrity of the characters. Contour analysis is another useful technique that can extract the contour information of characters, helping to determine the position and shape of the characters. This is crucial for ensuring that each local image contains only text from a single language.

[0044] Since the language of the text has already been determined using a language-determining neural network, the image can be further segmented based on this language information. If the image contains multiple languages, it should be ensured that the text in each language is segmented into a separate local image. During the segmentation process, multiple iterations and optimizations may be required to ensure that each local image meets the requirements. This may involve fine-tuning the segmentation algorithm or adjusting the processing logic for specific situations.

[0045] Furthermore, after dividing the image to be identified into multiple local images, the method further includes embedding special characters representing a specific language before and after each local image to indicate the text language of each local image.

[0046] Specifically, first, a special and infrequently used character or symbol needs to be selected or designed for each target language. These special characters should not conflict with any characters used in the text content. For example, rare Unicode characters or custom symbols can be used. Next, determine the specific embedding positions of these special characters in the local image. Usually, these characters are placed at the beginning and end of the text line, that is, the first and last character positions of each local image. The advantage of doing this is that the language of each text line can be quickly identified without disturbing the normal text content. Embed the selected special characters before and after each local image. This can be achieved through image editing software or programming tools to ensure that the special characters are clearly distinguishable from the text in the local image. The embedded characters should match the text style and font in the local image to maintain visual consistency. When embedding special characters, attention should be paid to preserving the original visual features of the local image, such as color, contrast, and brightness. This helps the subsequent OCR system to more accurately identify and process these images.

[0047] After embedding the special characters, generate a new marked image that contains the original local image and the added special characters. This marked image will be used as the input for subsequent OCR recognition and text processing. For ease of subsequent processing, the language information of each local image, that is, the language represented by the special characters, should also be recorded in a separate data structure. This will help to automatically identify and process texts in different languages.

[0048] To better illustrate the principle of the above implementation method, a specific example is given below.

[0049] Suppose the input text line image contains the following content: "我爱中国 this english prüfen". The input image contains Chinese "我爱中国", English "this", and German "english prüfen". Then use the Transformer encoder to extract the image features, and use MLP + sigmoid and MLP + softmax to judge the text position and language. Output the position probabilities, for example: [0.1, 0.9, 0.2, 0.8, 0.6, 0.4], where 1 indicates the presence of text and 0 indicates the background. Output the language probability distribution, for example: [[0.9, 0.05, 0.03, 0.02], [0.05, 0.9, 0.03, 0.02], [0.02, 0.03, 0.9, 0.05], [0.02, 0.03, 0.05, 0.9]], where 0 represents Chinese, 1 represents English, 2 represents German, and 3 represents others. Based on the text position and text language, the input is segmented into three local images: "我爱中国", "this", and "english prüfen".

[0050] Then, set special characters. <c> 、 <e>and <g>C represents Chinese, E represents English, and G represents German. After embedding special characters into each local image, the generated new labeled image is: <c>Chinese partial image <c> <e>English part of the image <e> <g>German part of the image <g>The marked image will serve as input for subsequent OCR recognition and text processing.

[0051] Step S25: After preprocessing the segmented local image, CTC technology is used to decode it to obtain the final text recognition result.

[0052] In some embodiments, the preprocessing includes steps S31-S34, such as... Figure 3 The diagram shows a flowchart of the preprocessing process.

[0053] Step S31: Use a multilayer perceptron network to extract special character features of each local image, and use a visual model to obtain the visual features of each local image.

[0054] Specifically, a Multilayer Perceptron (MLP) is a feedforward artificial neural network model capable of handling complex data relationships. In special character feature extraction, the MLP learns complex patterns and features of special characters through its multiple neuron layers. These special characters are often used to represent the language of text, so accurately extracting their features is crucial for subsequent language recognition and text processing. Each neuron performs linear transformations and non-linear activation operations, where the choice of activation function (such as ReLU or Sigmoid) significantly impacts network performance.

[0055] Visual models are typically convolutional neural networks (CNNs), specifically designed for processing and analyzing visual information. When acquiring visual features from local images, CNNs automatically learn useful features from the data through their convolutional, pooling, and fully connected layers. Convolutional layers capture local features, pooling layers reduce feature dimensionality and improve spatial invariance, and fully connected layers map these features to the final output category. This process helps extract effective visual features from images, providing the necessary information for text recognition.

[0056] Step S32: Concatenate the special character features and the visual features of the local image to form a new input matrix.

[0057] Specifically, the special character features extracted by the MLP and the visual features acquired by the CNN are concatenated to form a new input matrix. This matrix integrates information from different sources, providing rich feature representations for subsequent decoding steps.

[0058] Step S33: Encode the new input matrix using a neural network language model to generate a feature matrix.

[0059] Specifically, the new input matrix is ​​encoded using a neural network language model to generate a feature matrix. This step transforms the input matrix into a fixed-length sequence containing key information for text recognition.

[0060] Step S34: Input the feature matrix into the CTC decoder for decoding to obtain the final text recognition result.

[0061] Specifically, the CTC decoder is a decoding technique specifically designed for processing sequential data. It can directly predict the most likely character sequence from the input feature matrix. The CTC decoder considers all possible alignment methods, thus achieving accurate text recognition without using alignment information.

[0062] The above embodiments, through preprocessing and CTC decoding, can effectively obtain the final text recognition result from the segmented local images. This process integrates multiple key steps such as special character feature extraction, visual feature acquisition, feature concatenation, feature matrix encoding, and CTC decoding, achieving accurate recognition and processing of multilingual text.

[0063] Continuing with the example of "I love China this english prüfen" above, let's introduce the subsequent steps in "Generating a New Markup Image". For example... Figure 4 The diagram shown illustrates the complete steps of this example. A multilayer perceptron network (MLP) is used to extract... <c>Chinese partial image <c> 、 <e>English part of the image <e> 、 <g>German part of the image <g>The special character features were obtained using a visual model (VIT). <c>Chinese partial image <c> <e>English part of the image <e> <g>German part of the image <g>Visual features. Concatenate the obtained special character features and the visual features of the local image to form a new input matrix. Encode the new input matrix through a neural network language model (LLM) to generate a feature matrix. Finally, input the feature matrix into a CTC decoder for decoding to obtain the final text recognition result, "I love China this english prüfen".

[0064] Further, after obtaining the final text recognition result, as Figure 5 shown, a schematic diagram of the post-processing step flow is presented, including steps S51~S52:

[0065] Step S51: Match the decoded text with a preset special character library to determine and separate the text content and corresponding language information of each local image.

[0066] Step S52: According to the separated language information, convert the text content into a unified representation format or translate it into the target language for subsequent use.

[0067] Specifically, match the decoded text with a preset special character library. The special character library contains all possible special characters and their corresponding language information. The purpose of the match is to determine the text content and corresponding language information of each local image. Through the match, the text content and corresponding language information of each local image are separated. For example, if the special character "" is matched, it is determined that the local image contains Chinese text. According to the separated language information, convert the text content into a unified representation format or translate it into the target language. For example, if it is necessary to convert Chinese text into English, machine translation technology can be used to translate Chinese into English. If translation is not required, convert the text content into a unified representation format for subsequent use. For example, convert the text content to lowercase, remove punctuation marks, and perform unified word segmentation, etc.

[0068] By matching the decoded text with the special character library, separating the text content and language information, and converting it into a unified representation format or translating it into the target language, it can effectively support the subsequent use of a multilingual OCR system. This process not only improves the usability of text data but also promotes cross-language information sharing and application development.

[0069] It should be specifically noted that the flowchart representations of the embodiments described above in this disclosure can be understood as representing modules, segments, or portions of code comprising one or more sets of executable instructions configured to implement specific logical functions or processes. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved.

[0070] like Figure 5 As shown, the multilingual text recognition device 60 of this disclosure is illustrated. It should be noted that the principle and technical implementation of the multilingual text recognition device can refer to the multilingual text recognition method embodiments in the previous embodiments (e.g., Figure 1 Therefore, this embodiment will not repeat the details.

[0071] Specifically, the multilingual text recognition device 60 includes: an acquisition module 61, a feature extraction module 62, a judgment module 63, a segmentation module 64, and a recognition module 65, wherein,

[0072] The acquisition module 61 is used to acquire an image to be recognized that contains text in multiple languages.

[0073] The feature extraction module 62 is used to extract features from the image to be identified using a Transformer encoder to obtain image features.

[0074] The judgment module 63 is used to determine the text position of the image to be identified by a position judgment neural network based on the image features, and to determine the language of the text by a language judgment neural network.

[0075] The segmentation module 64 is used to segment the image to be identified into multiple local images according to the text location and the text language, and each local image contains text in a single language.

[0076] The recognition module 65 is used to preprocess the segmented local image and then decode it using CTC technology to obtain the final text recognition result.

[0077] It should be noted that, Figure 6 The various functional modules in the embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a program instruction product. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0078] and, Figure 6 The apparatus disclosed in the embodiments can be implemented through other modular division methods. The apparatus embodiments shown above are merely illustrative. For example, the module division is only a logical functional division, and in actual implementation, there may be other division methods. For example, a group of modules or modules may be combined or dynamically integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces, and the indirect coupling or communication connection between devices or modules may be electrical or other forms.

[0079] in addition, Figure 6 The functional modules and sub-modules in the embodiments can be dynamically integrated within a single processing unit, or each module can exist physically independently, or two or more modules can be dynamically integrated within a single unit. These dynamic units can be implemented in hardware or as software functional modules. If these dynamic units are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0080] like Figure 7 The diagram shown illustrates the structure of an electronic device according to an embodiment of this disclosure.

[0081] The electronic device can execute computer program instructions to perform tasks such as... Figure 1 The method in any of the above. For example, the electronic device may be a server group / server, a desktop computer, a laptop computer, etc., or a cloud-based server / server group, a distributed computing node system, etc., that communicates remotely with a local terminal.

[0082] The electronic device 70 includes a bus 71, a processor 72, and a memory 73. The processor 72 and the memory 73 can communicate via the bus 71. The memory 73 can store program instructions. The processor 72 implements the method steps in the previous embodiments by executing the program instructions in the memory 73, such as... Figure 1 Any one of the methods.

[0083] Bus 71 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, although only one thick line is used in the diagram, this does not indicate that there is only one bus or one type of bus.

[0084] In some embodiments, processor 72 may be implemented as a central processing unit (CPU), microprocessor unit (MCU), system on chip (System on Chip), or field-programmable array (FPGA). Memory 73 may include volatile memory for temporary data storage during program execution, such as random access memory (RAM).

[0085] The memory 73 may also include non-volatile memory for data storage, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state disk (SSD).

[0086] In some embodiments, the electronic device 70 may further include a communicator 74. The communicator 74 is used for communicating with external devices. In specific examples, the communicator 74 may include one or more wired and / or wireless communication circuit modules. For example, the communicator 74 may include one or more of, such as a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, one or more of the following: Nearfield Communication (NFC) technology, Infrared (IR) technology, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), and Global Navigation Satellite System (GNSS).

[0087] This disclosure also provides a computer-readable storage medium, characterized in that it stores program instructions, which are executed, for example... Figure 1 The multilingual text recognition method in the embodiments.

[0088] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium after being downloaded via a network, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a special processor or programmable or special hardware (such as ASIC or FPGA).

[0089] In summary, the multilingual text recognition method, apparatus, device, and medium disclosed herein aim to improve the performance and accuracy of multilingual text recognition by introducing improved structures of position-determining neural networks and language-determining neural networks, as well as optimized segmentation and preprocessing strategies for the images to be recognized. Furthermore, it involves embedding special characters representing specific languages ​​during the recognition process, and using multilayer perceptron networks and visual models to extract features, further obtaining the final text recognition result through a CTC decoder, which can more accurately indicate the language of the text in each local image.

[0090] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.< / g> < / g> < / e> < / e> < / c> < / c> < / g> < / g> < / e> < / e> < / c> < / c> < / g> < / g> < / e> < / e> < / c> < / c> < / g> < / e> < / c>

Claims

1. A method for recognizing multilingual text, characterized in that, include: Acquire images containing text in multiple languages ​​to be recognized; The Transformer encoder is used to extract features from the image to be identified in order to obtain image features; Based on the image features, the text position of the image to be identified is determined by a position determination neural network, and the language of the text is determined by a language determination neural network. Based on the text location and the text language, the image to be identified is segmented into multiple local images, each of which contains text in a single language; After preprocessing the segmented local images, CTC technology is used for decoding to obtain the final text recognition result.

2. The multilingual text recognition method according to claim 1, characterized in that, Also includes: The input to the location determination neural network is a subset of the image features, which is used to determine the text location. The output of the location determination neural network is the probability that text exists at each pixel location; When the probability of the presence of text is greater than a threshold, the corresponding pixel position is determined as the text position.

3. The multilingual text recognition method according to claim 1, characterized in that, Also includes: The input to the language identification neural network is the language seed set of the image features, and the language seed set is used to identify the language of the text. The language identification neural network outputs a text language probability distribution. The language with the highest probability in the language probability distribution is determined as the language at the corresponding pixel position.

4. The multilingual text recognition method according to claim 1, characterized in that, include: The location determination neural network adopts a multilayer perceptron network, and its output layer uses the sigmoid activation function; The language identification neural network employs a multilayer perceptron network, and its output layer uses the softmax activation function.

5. The multilingual text recognition method according to claim 1, characterized in that, After segmenting the image to be identified into multiple local images, the process further includes: Special characters representing a specific language are embedded before and after each of the local images to indicate the text language of each local image.

6. The multilingual text recognition method according to claim 5, characterized in that, The segmented local images are preprocessed and then decoded using CTC technology to obtain the final text recognition results, including: The special character features of each local image are extracted using a multilayer perceptron network, and the visual features of each local image are obtained using a visual model. The special character features and the visual features of the local image are concatenated to form a new input matrix; The new input matrix is ​​encoded using a neural network language model to generate a feature matrix; The feature matrix is ​​input into the CTC decoder for decoding to obtain the final text recognition result.

7. The method for recognizing multilingual text according to claim 1, characterized in that, After obtaining the final text recognition result, the process also includes: The decoded text is matched with a preset special character library to determine and separate the text content and corresponding language information of each local image; Based on the extracted language information, the text content is converted into a unified representation format or translated into the target language for subsequent use.

8. A multilingual text recognition device, characterized in that, include: The acquisition module is used to acquire images containing text in multiple languages ​​to be recognized; The feature extraction module is used to extract features from the image to be identified using a Transformer encoder to obtain image features; The judgment module is used to determine the text position of the image to be recognized by means of a position judgment neural network based on the image features, and to determine the language of the text by means of a language judgment neural network. The segmentation module is used to segment the image to be identified into multiple local images based on the text location and the text language, each local image containing text in a single language; The recognition module is used to preprocess the segmented local images and then decode them using CTC technology to obtain the final text recognition result.

9. An electronic device, characterized in that, The electronic device includes: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to cause the electronic device to perform the multilingual text recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the multilingual text recognition method according to any one of claims 1 to 7.

Images