Text recognition method and device, equipment and medium

Abstract

The embodiment of the invention relates to a text recognition method and device, equipment and a medium. The method comprises the steps that a to-be-recognized text image is acquired; inputting the to-be-recognized text image into a multi-modal recognition model for recognition to obtain a known character result and a first unknown character result; extracting an unknown character image from the to-be-recognized text image based on the first unknown character result; inputting the unknown character image into the character recognition model to obtain a second unknown character result; and determining the known character result and the second unknown character result as an identification result of the to-be-identified text image. According to the method, after the unknown characters are positioned through the multi-modal recognition model, the character recognition model is further utilized to correct the unknown characters, the capabilities of the large and small models are fully combined, accurate recognition of the unknown characters is effectively realized, the accuracy and robustness of text recognition can be ensured, and the training cost of model optimization can be effectively reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a text recognition method, device, equipment and medium. Background Art

[0002] Text recognition is a research field that integrates visual perception and language understanding, involving tasks such as detecting and recognizing characters from images containing text, understanding text semantics, and correcting character positioning. In related technologies, text recognition is usually based on traditional optical character recognition (OCR) technology or implemented using multimodal models. However, traditional technologies have problems with low accuracy and robustness, and multimodal models can only recognize rare or unknown characters through retraining, which is costly. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a text recognition method, device, equipment and medium.

[0004] The present disclosure provides a text recognition method, the method comprising:

[0005] Get the text image to be recognized;

[0006] Inputting the to-be-recognized text image into a multimodal recognition model for recognition, and obtaining a known character result and a first unknown character result;

[0007] Extracting an unknown character image from the to-be-recognized text image based on the first unknown character result;

[0008] Inputting the unknown character image into a character recognition model to obtain a second unknown character result;

[0009] The known character result and the second unknown character result are determined as recognition results of the text image to be recognized.

[0010] The present disclosure also provides a text recognition device, the device comprising:

[0011] An acquisition module, used for acquiring a text image to be recognized;

[0012] A first recognition module, used for inputting the to-be-recognized text image into a multimodal recognition model for recognition, and obtaining a known character result and a first unknown character result;

[0013] An extraction module, used for extracting an unknown character image from the to-be-recognized text image based on the first unknown character result;

[0014] A second recognition module, used for inputting the unknown character image into a character recognition model to obtain a second unknown character result;

[0015] A result module is used to determine the known character result and the second unknown character result as the recognition result of the text image to be recognized.

[0016] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the text recognition method provided by the embodiment of the present disclosure.

[0017] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the text recognition method provided by the embodiment of the present disclosure.

[0018] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: the text recognition solution provided by the embodiment of the present disclosure obtains a text image to be recognized; inputs the text image to be recognized into a multimodal recognition model for recognition, and obtains a known character result and a first unknown character result; extracts an unknown character image from the text image to be recognized based on the first unknown character result; inputs the unknown character image into a character recognition model, and obtains a second unknown character result; and determines the known character result and the second unknown character result as the recognition result of the text image to be recognized. By adopting the above technical solution, the image is first recognized by a multimodal recognition model to obtain a known character result and a first unknown character result, and the unknown character image extracted based on the first unknown character result can be further recognized by a character recognition model to obtain a second unknown character result, and after locating the unknown character by the multimodal recognition model, the character recognition model is further used to correct the unknown character, and the capabilities of the large and small models are fully combined to effectively realize the accurate recognition of unknown characters, and the accuracy and robustness of text recognition can be guaranteed, and the training cost of model optimization can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of a text recognition method provided in some embodiments of the present disclosure;

[0021] Figure 2 A schematic diagram of a text recognition process provided for some embodiments of the present disclosure;

[0022] Figure 3 A schematic diagram of a model training process provided for some embodiments of the present disclosure;

[0023] Figure 4 A schematic diagram of another model training process provided for some embodiments of the present disclosure;

[0024] Figure 5 A schematic diagram of the structure of a text recognition device provided in some embodiments of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided for some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] In the related art, one way of text recognition is to implement it based on traditional OCR technology. Traditional OCR technology is usually based on Convolutional Neural Network (CNN) combined with classification layer for character recognition. However, this solution has obvious limitations when processing complex text scenes. Its model capability is mainly limited to the pre-defined character set, resulting in insufficient recognition of unknown characters or long-tail characters. In addition, the fixed receptive field of the convolution layer and the limitation of the classification layer make it difficult for the OCR solution to adapt to diverse character styles and complex text arrangements, affecting the accuracy and robustness of recognition. Another way of text recognition is to implement it using multimodal model technology. Multimodal models have strong capabilities in processing image and text information fusion, but when facing uncommon characters, it is difficult to output correct results. Due to the extremely low frequency of occurrence of uncommon characters, it is difficult to effectively process these characters by simply expanding the vocabulary, or it is easy to recognize them as similar characters, resulting in reduced accuracy. Vocabulary expansion not only increases the complexity and training cost of the model, but may also lead to insufficient model training due to data scarcity, thereby affecting the recognition and understanding of unknown characters.

[0033] In order to solve the above problems, the embodiments of the present disclosure provide a text recognition method, which is introduced below in conjunction with specific embodiments.

[0034] Figure 1 The flowchart of the text recognition method provided by some embodiments of the present disclosure is shown in FIG. 1 , which can be performed by a text recognition device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:

[0035] Step 101: Obtain a text image to be recognized.

[0036] Among them, the text image to be recognized can be any image that requires text recognition. For example, the target text image can include a document image, a chat screenshot, a card image, a bill image, etc. The specific type and quantity are not limited and can be set according to actual conditions.

[0037] The text recognition device can obtain a text image to be recognized input by a user. The source of the text image to be recognized is not limited, and can be obtained by real-time photography or obtained from the Internet. This is only an example.

[0038] Optionally, after obtaining the text image to be recognized, the text recognition method may further include: adjusting the original size of the text image to be recognized to a preset size, which may be set according to actual conditions. For example, the size of the text image to be recognized may be adjusted from the original size to 384x384, and in order to prevent image distortion, the larger size of the width and height may be adjusted to 384, and the other may be scaled proportionally and then filled with 0 values ​​to 384. The size of the text image to be recognized is unified through the preprocessing of size adjustment, which may be beneficial to the subsequent detection and recognition of the image and improve efficiency.

[0039] Step 102: input the text image to be recognized into the multimodal recognition model for recognition, and obtain a known character result and a first unknown character result.

[0040] Among them, the multimodal recognition model can be a model for text content recognition and text position recognition of an image, and can detect each character included and the corresponding character position. The multimodal recognition model can be obtained based on large-scale data pre-training, and can be obtained through a series of sub-task pre-training based on massive data (text, image, video, etc.), and has a strong content understanding ability and high accuracy in image recognition. In the disclosed embodiment, the multimodal recognition model is trained by the first training data constructed by the basic word list. The basic word list can be a word list that summarizes commonly used words. Commonly used words can be frequently used and widely recognized words, usually including nouns, verbs, adjectives, adverbs, prepositions, conjunctions and other parts of speech, covering words required for communication, writing, learning, work and other aspects. The multimodal recognition model trained by the basic word list can accurately recognize commonly used words.

[0041] The known character result may be an accurate result obtained by recognizing a word in the basic word list included in the text image to be recognized. The known character result may include the character content and character position of the known character. The known character represents the character whose content can be accurately recognized by the multimodal recognition model in the text image to be recognized. The first unknown character result may be the result of recognizing an unknown character that is not in the basic word list and included in the text image to be recognized using the multimodal recognition model. The first unknown character result does not recognize the content of the unknown character but only extracts the unknown character. The unknown character represents the character that cannot be recognized by the multimodal recognition model in the text image to be recognized. The unknown character may include long-tail characters, uncommon characters, etc. Long-tail characters may be characters that appear less frequently and have a narrower range of use than common characters in a certain context or field. Uncommon characters may be characters that rarely appear in daily life and general texts, or even characters that are not included in some standard character sets. The first unknown character result includes a unified identifier and character position of the unknown character. The unified identifier may be a specific identifier (token) that represents the unknown character. In the embodiment of the present disclosure, all unknown characters are represented by the unified identifier. The unified identifier may be, for example, represented by " <unk>" indicates that the number of unknown characters can be at least one.

[0042] Specifically, the text recognition device can input the text image to be recognized into the multimodal recognition model for recognition. For example, the visual encoder of the multimodal recognition model can extract the corresponding visual unit feature sequence from the text image to be recognized, and then adjust the dimension of the feature through the bridging module to align it with the hidden layer feature dimension of the multimodal recognition model. Subsequently, it is passed to the text decoder of the model, and the final text detection and recognition result, that is, the known character result and the first unknown character result, is output through the attention mechanism of the model. The character position in the result can be represented by the coordinates of the four vertices of the corresponding rectangular area of ​​the character or the upper left and lower right coordinates.

[0043] Step 103: extract an unknown character image from the text image to be recognized based on the first unknown character result.

[0044] The unknown character image may be a partial image of the region corresponding to the unknown character in the text image to be recognized. After determining the first unknown character result, the text recognition device may crop the region image at the character position of the unknown character in the text image to be recognized according to the position of the unknown character in the first unknown character result to obtain the unknown character image corresponding to the unknown character, where the region image may be a rectangular region image selected in the text image to be recognized according to the coordinates of the character position.

[0045] Step 104: input the unknown character image into the character recognition model to obtain a second unknown character result.

[0046] The character recognition model can be a model for recognizing uncommon words in an image, and can recognize long-tail characters, uncommon characters and other uncommon words. The character recognition model is a lightweight recognition model, which is used to perform the next stage of recognition tasks for unknown characters in the above embodiment. Since the input of the model is a specific single character, it has the characteristics of relatively simple tasks, easy training, and high reasoning efficiency, and can be fine-tuned using a general visual recognition model. The character recognition model in the disclosed embodiment is trained with the second training data constructed by a non-basic word vocabulary. The non-basic word vocabulary can be a word list that summarizes uncommon words. The common words can be removed from the full word list, and the remaining characters can be used as the non-basic word vocabulary. Uncommon words can be other words other than the above-mentioned basic word vocabulary, and can include long-tail characters, uncommon characters, etc. The character recognition model can be a classification model that uses the number of non-basic words in the non-basic word vocabulary as a classifier dimension.

[0047] Specifically, the text recognition device can input an unknown character image into a character recognition model for recognition to obtain a second unknown character result. The second unknown character result can be the result of using the character recognition model to recognize an unknown character included in the text image to be recognized that is not in the basic word list. The second unknown character result accurately identifies the content of the unknown character to obtain its specific text content. The second unknown character result includes the character content of the unknown character, and the character position of the unknown character is known from the first unknown character result.

[0048] Since the character recognition model is a lightweight model, when the non-basic word list needs to be expanded to support additional characters, it is only necessary to simply modify the classification layer of the recognition model and retrain or incrementally train it. This link can effectively reduce the training cost. Compared with the training of the multimodal recognition model, it has the characteristics of small training volume and high efficiency, and can achieve rapid iteration.

[0049] Step 105: Determine the known character result and the second unknown character result as the recognition result of the text image to be recognized.

[0050] Specifically, after obtaining the known character result and the second unknown character result, the text recognition device can combine the known character result and the second unknown character result to determine the recognition result of the text image to be recognized. The recognition result can be the precise text content and text unknown of the text recognized in the text image to be recognized.

[0051] For example, Figure 2 A schematic diagram of a text recognition process provided by some embodiments of the present disclosure, such as Figure 2 As shown in the figure, the text image to be recognized is an image including English characters as an example. The specific process may include: the multimodal recognition model is trained by the first training data of the basic word list, and then the text image to be recognized is input into the multimodal recognition model for recognition to obtain a known character result and a first unknown character result. The known character result includes the character content and character position of the known character, and the character position is represented by coordinates. The first unknown character result includes a unified identifier and a character position of the unknown character. In the figure, the text image to be recognized includes only one unknown character "é", and the unified identifier is represented by " <unk>" represents; according to the character position of the unknown character in the first unknown character result, an unknown character image can be extracted from the text image to be recognized, and the unknown character image is input into the character recognition model, and a second unknown character result is output. The character recognition model is trained by the second training data of the non-basic word list; finally, the known character result and the second unknown character result are combined to determine the recognition result of the text image to be recognized. As shown in the figure, for the word "café" where the unknown character is located, the character content "é" of the unknown character in the second unknown character result is filled in to realize the correction of the characters that the multimodal recognition model fails to recognize.

[0052] This solution is designed for text recognition scenarios based on a combination of a multimodal recognition model and a traditional character recognition model. Based on the character positioning capability of the multimodal recognition model, a lightweight character recognition model is used to correct unknown characters, thereby achieving accurate text understanding tasks. In addition, for unknown characters in the text, an automatic data generation solution is designed to train additional character recognition models. By fully combining the capabilities of large and small models, the training cost of solution optimization is effectively reduced, which can solve the recognition problem of unknown characters or long-tail characters.

[0053] The text recognition scheme provided by the embodiment of the present disclosure obtains a text image to be recognized; inputs the text image to be recognized into a multimodal recognition model for recognition to obtain known character results and a first unknown character result; extracts an unknown character image from the text image to be recognized based on the first unknown character result; inputs the unknown character image into a character recognition model to obtain a second unknown character result; and determines the known character result and the second unknown character result as the recognition result of the text image to be recognized. Using the above technical scheme, the image is first recognized by a multimodal recognition model to obtain a known character result and a first unknown character result, and the unknown character image extracted based on the first unknown character result can be further recognized by a character recognition model to obtain a second unknown character result. After locating the unknown character by the multimodal recognition model, the character recognition model is further used to correct the unknown character, and the capabilities of the large and small models are fully combined to effectively realize the accurate recognition of unknown characters, and the accuracy and robustness of text recognition can be guaranteed, while the training cost of model optimization can be effectively reduced.

[0054] In some embodiments, the text recognition method may also include: obtaining a first sample text image and a sample known character result corresponding to the first sample text image determined based on a basic word vocabulary; randomly replacing non-basic words in the first sample text image to generate a second sample text image; and constructing first training data based on the second sample text image, the sample known character result, and the sample unknown character result of non-basic words.

[0055] Among them, the first sample text image can be a large-scale image with text generated for the multimodal recognition model, and the sample known character result can be a labeling result determined for the first sample text image according to the basic word list, which can include the character content and character position of the recognized sample known characters. The second sample text image can be an image including at least one non-basic word generated by character replacement of the first sample text image. The sample unknown character result can include the non-basic word as a unified identifier and character position corresponding to the unknown character in the second sample text image, which can be determined when the second sample text image is generated. The first training data can be training data for training the multimodal recognition model, and the data volume is large.

[0056] The text recognition device can first construct the first training data for the multimodal recognition model. Specifically, one or more characters in the first sample text image can be randomly replaced with non-basic words, and the replaced first sample text image is called the second sample text image. Then, the second sample text image is used as input data, the sample known character result of the first sample text image, and the sample unknown character result of the second sample text image is used as the annotation result to construct the first training data. Then, the first training data can be used to train the initial model, and the trained initial model is determined as the multimodal recognition model. The multimodal recognition model will no longer be retrained during subsequent use to reduce costs.

[0057] For example, Figure 3 A schematic diagram of a model training process provided for some embodiments of the present disclosure, such as Figure 3 As shown in the figure, the training process of the multimodal recognition model is shown. Specifically, the multimodal recognition model can be obtained by training the first training data constructed by the basic word list.

[0058] In the above scheme, the training data for the multimodal recognition model can be obtained by first generating a large number of images with text, and then randomly replacing non-basic words in the images with text so that they contain a certain amount of unknown characters, thereby improving their text recognition ability and the ability to detect unknown character areas. The character replacement scheme is used to realize automatic generation of training data to improve data diversity and training robustness.

[0059] In some embodiments, the text recognition method may further include: enhancing the non-basic word images corresponding to the non-basic word vocabulary to obtain enhanced images; and constructing second training data based on the non-basic word images, enhanced images and corresponding non-basic words.

[0060] Among them, the non-basic word image can be an image including only one non-basic word, and all non-basic words included in the non-basic word vocabulary have corresponding non-basic word images. The enhancement processing can be a deformation processing performed on the non-basic word image, and the enhancement processing can include rotation, scaling, adding noise, blurring, etc., which are used to increase the diversity of data. The enhanced image can be an image obtained after the non-basic word image is enhanced, and one non-basic word image can correspond to one or more enhanced images. The second training data can be data for training a character recognition model, and the second training data can be reconstructed when the non-basic word vocabulary is expanded, and the character recognition model can be retrained to be able to recognize the expanded non-basic words.

[0061] Specifically, the text recognition device can perform enhancement processing on the non-basic word image of each non-basic word corresponding to the non-basic word vocabulary to obtain an enhanced image. A non-basic word image can be processed accordingly to obtain one or more enhanced images. Then, the non-basic word image and the corresponding non-basic word can be used as one data, and an enhanced image and the corresponding non-basic word can be used as one data to combine and construct second training data. Then, the second training data can be used to train the neural network. During training, the non-basic word image or the enhanced image is used as input data, and the corresponding non-basic word is trained as the annotation result. The trained neural network is determined as a character recognition model. When the non-basic word vocabulary needs to be expanded to support additional characters later, it is only necessary to simply modify the classification layer of the recognition model, and retrain or incrementally train. This link can effectively reduce the training cost.

[0062] For example, Figure 4 A schematic diagram of another model training process provided for some embodiments of the present disclosure, such as Figure 4 As shown in the figure, the training process of the character recognition model is shown. Specifically, the non-basic word images corresponding to the expanded non-basic word vocabulary can be enhanced to obtain enhanced images, and then the second training data can be constructed. The character recognition model can be trained using the second training data.

[0063] In the above scheme, by enhancing the non-basic word images to increase the diversity of the training data of the character recognition model, not only the robustness of the character recognition model under different conditions is improved, but also its recognition ability for rare characters can be effectively improved.

[0064] The text recognition solution provided by the embodiments of the present disclosure can effectively solve the recognition problem of unknown characters and long-tail characters in text recognition scenarios; by integrating and coordinating the multimodal model and the lightweight character recognition model, it can simultaneously achieve a high recognition level and rapid expansion and fine-tuning capabilities; and, combined with the proposed technology, the automated training data generation solution for the two models designed by the present disclosure can achieve rapid iteration of training, implement low-cost optimization training for newly added unknown characters, and effectively reduce the training cost of solution optimization.

[0065] Figure 5 This is a schematic diagram of the structure of a text recognition device provided in some embodiments of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 5 As shown, the device comprises:

[0066] An acquisition module 501 is used to acquire a text image to be recognized;

[0067] A first recognition module 502 is used to input the to-be-recognized text image into a multimodal recognition model for recognition, and obtain a known character result and a first unknown character result;

[0068] An extraction module 503, configured to extract an unknown character image from the to-be-recognized text image based on the first unknown character result;

[0069] A second recognition module 504, used for inputting the unknown character image into a character recognition model to obtain a second unknown character result;

[0070] The result module 505 is used to determine the known character result and the second unknown character result as the recognition result of the text image to be recognized.

[0071] Optionally, the multimodal recognition model is trained using first training data constructed from a basic word vocabulary.

[0072] Optionally, the first unknown character result includes a unified identifier of the unknown character and a character position, wherein the number of the unknown characters is at least one.

[0073] Optionally, the extraction module 503 is used for:

[0074] The region image at the character position of the unknown character in the text image to be recognized is cropped to obtain an unknown character image corresponding to the unknown character.

[0075] Optionally, the character recognition model is trained using second training data constructed from a non-basic word vocabulary.

[0076] Optionally, the device further includes a first data module, configured to:

[0077] Performing enhancement processing on the non-basic word images corresponding to the non-basic word vocabulary to obtain enhanced images;

[0078] The second training data is constructed based on the non-basic word images, the enhanced images, and the corresponding non-basic words.

[0079] Optionally, the device further includes a second data module, configured to:

[0080] Acquire a first sample text image and a sample known character result corresponding to the first sample text image determined based on the basic word list;

[0081] Randomly replacing non-basic words in the first sample text image to generate a second sample text image;

[0082] The first training data is constructed based on the second sample text image, the sample known character results, and the sample unknown character results of the non-basic words.

[0083] The text recognition device provided in the embodiments of the present disclosure can execute the text recognition method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0084] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which implements the text recognition method provided by any embodiment of the present disclosure when executed by a processor.

[0085] Figure 6 This is a schematic diagram of the structure of an electronic device provided in some embodiments of the present disclosure. Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The electronic device 600 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0086] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0087] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0088] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the text recognition method of the embodiment of the present disclosure are executed.

[0089] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0090] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0091] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0092] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the conference text of the conference audio and video; inputs the conference text into the to-do recognition model to determine the initial to-do statement; inputs the initial to-do statement into the tense judgment model to determine the tense result; and determines the conference to-do statement in the initial to-do statement based on the tense result.

[0093] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0096] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0097] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0099] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0100] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0101] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.< / unk> < / unk>

Claims

1. A text recognition method, characterized in that: include: Get the text image to be recognized; Inputting the to-be-recognized text image into a multimodal recognition model for recognition, and obtaining a known character result and a first unknown character result; Extracting an unknown character image from the to-be-recognized text image based on the first unknown character result; Inputting the unknown character image into a character recognition model to obtain a second unknown character result; The known character result and the second unknown character result are determined as recognition results of the text image to be recognized.

2. The method according to claim 1, characterized in that The multimodal recognition model is trained by first training data constructed through a basic word vocabulary.

3. The method according to claim 1, characterized in that: The first unknown character result includes a unified identifier of the unknown character and a character position, wherein the number of the unknown characters is at least one.

4. The method according to claim 3, characterized in that Extracting an unknown character image from the to-be-recognized text image based on the first unknown character result includes: The region image at the character position of the unknown character in the text image to be recognized is cropped to obtain an unknown character image corresponding to the unknown character.

5. The method according to claim 1, characterized in that The character recognition model is trained by second training data constructed through a non-basic word vocabulary.

6. The method according to claim 5, characterized in that The method further comprises: Performing enhancement processing on the non-basic word images corresponding to the non-basic word vocabulary to obtain enhanced images; The second training data is constructed based on the non-basic word images, the enhanced images, and the corresponding non-basic words.

7. The method according to claim 2, characterized in that: The method further comprises: Acquire a first sample text image and a sample known character result corresponding to the first sample text image determined based on the basic word list; Randomly replacing non-basic words in the first sample text image to generate a second sample text image; The first training data is constructed based on the second sample text image, the sample known character results, and the sample unknown character results of the non-basic words.

8. A text recognition device, characterized in that: include: An acquisition module, used for acquiring a text image to be recognized; A first recognition module, used for inputting the to-be-recognized text image into a multimodal recognition model for recognition, and obtaining a known character result and a first unknown character result; An extraction module, used for extracting an unknown character image from the to-be-recognized text image based on the first unknown character result; A second recognition module, used for inputting the unknown character image into a character recognition model to obtain a second unknown character result; A result module is used to determine the known character result and the second unknown character result as the recognition result of the text image to be recognized.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the text recognition method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the text recognition method described in any one of claims 1 to 7.

Images