Method, device and equipment for recognizing OCR text based on hotword perception and medium

Abstract

The present disclosure relates to a hotword-aware OCR text recognition method, device, equipment and medium. The hotword-aware OCR text recognition method comprises: acquiring an image comprising text to be recognized, performing optical character recognition on the image to obtain first context information; acquiring a hotword library, the hotword library comprising hotwords, and perceiving a target hotword, the target hotword being determined in the hotword library to match the text to be recognized in the image, and determining context information containing the target hotword to obtain second context information; fusing the first context information and the second context information to obtain fused context information; and obtaining a text recognition result based on the fused context information. Through the present disclosure, the text recognition method has the ability to perceive hotwords, and the accuracy of text recognition is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, and in particular to OCR text recognition methods, apparatus, devices and media based on hot word perception. Background Technology

[0002] With the development of Internet technology, Optical Character Recognition (OCR) technology is also constantly improving. It eliminates the need for manual entry of paper documents. By uploading images, scanned copies, or taking real-time photos, it can accurately convert image text into editable text within seconds. Furthermore, batch processing is more efficient, saving a significant amount of time and doubling the efficiency of data entry and document archiving.

[0003] However, in related technologies, OCR recognition often only recognizes regular content, while treating hot words such as professional terms and jargon in images as the same as regular content. It lacks the ability to perceive hot words in different fields, which leads to a decrease in the accuracy of the final recognition results. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an OCR text recognition method, apparatus, device and medium based on hot word perception.

[0005] According to a first aspect of the present disclosure, an OCR text recognition method based on hot word awareness is provided, comprising:

[0006] An image containing the text to be recognized is acquired; optical character recognition is performed on the image to obtain first context information; a hot word library is acquired, which includes hot words, and a target hot word is perceived, wherein the perceived target hot word is a target hot word in the hot word library that matches the text to be recognized in the image, and context information containing the target hot word is determined to obtain second context information; the first context information and the second context information are fused to obtain fused context information; and a text recognition result is obtained based on the fused context information.

[0007] In one embodiment, the hot word library includes hot words, as well as visual and semantic features corresponding to the hot words; the perceived target hot word, which is the target hot word determined in the hot word library that matches the text to be identified in the image, includes:

[0008] Obtain the image feature sequence of the image during the optical character recognition process; determine the target hot words in the hot word library whose visual features and semantic features match the image feature sequence.

[0009] In one embodiment, determining target hot words in the hot word library whose visual and semantic features match the image feature sequence includes:

[0010] The similarity between the image feature sequence and the visual features of each hot word in the hot word library is determined to obtain visual similarity. The similarity between the image feature sequence and the semantic features of each hot word in the hot word library is also determined to obtain semantic similarity. The visual similarity and the semantic similarity are weighted to obtain the similarity between the image feature sequence and each hot word in the hot word library. Hot words whose similarity between the image feature sequence and each hot word in the hot word library is higher than a similarity threshold are selected as target hot words.

[0011] In one implementation, determining the context information of the target hot word to obtain second context information includes:

[0012] Based on the image feature sequence, the semantic and visual features of each character in the target hot word, the attention weights of the image features and each character in the target hot word are obtained; the attention weights of the image features and each character in the target hot word are normalized, and based on the normalized attention weights, the image feature sequence, the semantic and visual features of each character in the target hot word, the context information of the target hot word is obtained as the second context information.

[0013] In one implementation, fusing the first context information and the second context information to obtain fused context information includes:

[0014] Based on the first context information and the second context information, a gate value for feature fusion is determined; based on the gate value, the fusion ratio of the first context information and the second context information is dynamically adjusted, and fused context information is obtained based on the adjusted fusion ratio.

[0015] In one embodiment, the hot word library includes hot word libraries from multiple fields, and the process of obtaining the hot word library includes:

[0016] In response to obtaining the domain of the text to be identified, the hot word library matching the domain is used as the target hot word library; or in response to not obtaining the domain of the text to be identified, the hot word libraries of all domains in the plurality of domains are used as the target hot word library.

[0017] According to a second aspect of the present disclosure, an OCR text recognition device based on hot word perception is provided, comprising:

[0018] An acquisition unit is configured to acquire an image including the text to be recognized and acquire a hot word library, the hot word library including hot words; a processing unit is configured to perform optical character recognition on the image to obtain first context information and perceived target hot words, wherein the perceived target hot words are determined from the hot word library to match the text to be recognized in the image, and context information containing the target hot words is determined to obtain second context information, and the first context information and the second context information are fused to obtain fused context information, and a text recognition result is obtained based on the fused context information. In one embodiment, the hot word library includes hot words, as well as visual features and semantic features corresponding to the hot words; the processing unit determines the target hot words matching the text to be recognized in the image from the hot word library in the following manner:

[0019] Obtain the image feature sequence of the image during the optical character recognition process; determine the target hot words in the hot word library whose visual features and semantic features match the image feature sequence.

[0020] In one embodiment, the processing unit perceives target hot words by identifying target hot words in the hot word library whose visual and semantic features match the image feature sequence:

[0021] The similarity between the image feature sequence and the visual features of each hot word in the hot word library is determined to obtain visual similarity. The similarity between the image feature sequence and the semantic features of each hot word in the hot word library is also determined to obtain semantic similarity. The visual similarity and the semantic similarity are weighted to obtain the similarity between the image feature sequence and each hot word in the hot word library. Hot words whose similarity between the image feature sequence and each hot word in the hot word library is higher than a similarity threshold are selected as target hot words.

[0022] In one implementation, the processing unit determines the context information of the target hot word in the following manner to obtain second context information:

[0023] Based on the image feature sequence, the semantic and visual features of each character in the target hot word, the attention weights of the image features and each character in the target hot word are obtained; the attention weights of the image features and each character in the target hot word are normalized, and based on the normalized attention weights, the image feature sequence, the semantic and visual features of each character in the target hot word, the context information of the target hot word is obtained as the second context information.

[0024] In one embodiment, the processing unit fuses the first context information and the second context information in the following manner to obtain fused context information:

[0025] Based on the first context information and the second context information, a gate value for feature fusion is determined; based on the gate value, the fusion ratio of the first context information and the second context information is dynamically adjusted, and fused context information is obtained based on the adjusted fusion ratio.

[0026] In one embodiment, the hot word library includes hot word libraries from multiple fields, and the acquisition unit acquires the hot word library in the following manner:

[0027] In response to obtaining the domain of the text to be identified, the hot word library matching the domain is used as the target hot word library; or in response to not obtaining the domain of the text to be identified, the hot word libraries of all domains in the plurality of domains are used as the target hot word library.

[0028] According to a second aspect of the present disclosure, an OCR text recognition device based on hot word perception is provided, comprising:

[0029] An acquisition unit is used to acquire an image including text to be recognized; a recognition unit is used to perform optical character recognition on the image to obtain first context information; the acquisition unit is also used to acquire a hot word library, which includes hot words; a processing unit is used to determine a target hot word in the hot word library that matches the text to be recognized in the image, and determine the context information containing the target hot word to obtain second context information; a fusion unit is used to fuse the first context information and the second context information to obtain fused context information, and obtain a text recognition result based on the fused context information.

[0030] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the OCR text recognition method based on hot word perception as described in the first aspect or any embodiment of the first aspect.

[0031] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the OCR text recognition method based on hot word perception as described in the first aspect or any embodiment of the first aspect.

[0032] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: OCR recognition is performed on an image containing text to be recognized to obtain first contextual information; a hot word library is introduced to perceive target hot words matching the text to be recognized, and second contextual information containing the target hot words is determined; based on fused contextual information that integrates the first and second contextual information, a text recognition result is obtained. By combining the OCR recognition result and the hot word perception result, the influence of the hot word perception module result on the final recognition effect is dynamically adjusted, thereby improving the accuracy of the model's hot word perception for different domains while ensuring the accuracy of OCR recognition of text images.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0035] Figure 1 This is a flowchart illustrating an OCR text recognition method based on hot word awareness, according to an exemplary embodiment.

[0036] Figure 2 This is a flowchart illustrating a method for determining target hot words in a hot word library that match the text to be identified in an image, according to an exemplary embodiment.

[0037] Figure 3 This is a flowchart illustrating a method for determining target hot words in a hot word library that match visual features and semantic features with a sequence of image features, according to an exemplary embodiment.

[0038] Figure 4 This is a flowchart illustrating a method for determining the contextual information of a target hot word and obtaining second contextual information according to an exemplary embodiment.

[0039] Figure 5 This is a flowchart illustrating a method for fusing first context information and second context information to obtain fused context information, according to an exemplary embodiment.

[0040] Figure 6 This is a flowchart illustrating a method for obtaining text recognition results based on fused contextual information, according to an exemplary embodiment.

[0041] Figure 7 This is a flowchart illustrating a method for obtaining a hot word library according to an exemplary embodiment.

[0042] Figure 8This is a flowchart illustrating an OCR text recognition method based on hot word awareness, according to an exemplary embodiment.

[0043] Figure 9 This is a block diagram illustrating an OCR text recognition device based on hot word perception, according to an exemplary embodiment.

[0044] Figure 10 This is a block diagram illustrating an apparatus for OCR text recognition based on hot word awareness, according to an exemplary embodiment.

[0045] Figure 11 This is a block diagram illustrating an apparatus for OCR text recognition based on hot word awareness, according to an exemplary embodiment. Detailed Implementation

[0046] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0047] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0048] This disclosure provides some embodiments of an OCR text recognition method based on hot word awareness, which is applied in the field of computer vision technology.

[0049] In computer vision, Optical Character Recognition (OCR) is a technique that converts text in an image into editable text. Its development has evolved from traditional methods based on template matching and feature extraction to modern end-to-end systems based on deep learning. Traditional OCR relies on multi-stage processing (such as text detection, character segmentation, and single-character recognition), while modern OCR uses an encoder-decoder framework, directly inputting an image and outputting a text sequence, thus avoiding error propagation.

[0050] The encoder-decoder structure is the core framework of OCR. Key technologies include encoders such as Convolutional Neural Networks (CNNs) and Transformers to extract visual features, and decoders such as Connectionist Temporal Classification (CTC), Attention, or Transformer to generate character sequences, while combining language models to improve semantic consistency. The encoder is responsible for extracting multi-level visual features from the image, while the decoder converts the feature sequences into text. CTC is suitable for aligning sequences of variable length, the Attention mechanism achieves character-level generation by dynamically focusing on image regions, and the Transformer utilizes self-attention to model global dependencies. Extended technologies such as hot word awareness enhance the recognition of key information through dynamic dictionaries or language models, driving the development of OCR towards multimodal and adaptive scenarios.

[0051] However, in related technologies, when using OCR to recognize images, the images are often processed directly based on the OCR model. Traditional OCR models treat the content of the text to be recognized in the image equally, failing to focus on key words in the text that may exist in the field. They have low adaptability to professional fields and lack sensitivity to key words in different fields. When transferred to specific business scenarios, the accuracy drops, and the entire model needs to be retrained for different fields (legal / medical).

[0052] In view of this, some embodiments of this disclosure provide an OCR text recognition method based on hot word perception. In this text recognition method, a hot word library is introduced, and text recognition is performed by combining OCR recognition results and hot word perception results. While ensuring the accuracy of OCR recognition of text images, the accuracy of the model in perceiving hot words in different fields is improved.

[0053] Figure 1 This is a flowchart illustrating an OCR text recognition method based on hot word awareness, according to an exemplary embodiment, as follows: Figure 1 As shown, the method includes steps S101 to S104.

[0054] In step S101, an image including the text to be recognized is acquired, and optical character recognition is performed on the image to obtain the first context information.

[0055] In this embodiment of the disclosure, OCR recognition is the process by which an OCR model encodes and decodes the text to be recognized in an image. The image can be encoded using a CNN encoder, while the decoding process can be performed using a traditional decoder, such as a CTC decoder or an Attention decoder, to obtain the decoded first contextual information.

[0056] In step S102, a hot word library is obtained, which includes hot words, and a target hot word is perceived. The target hot word is determined by identifying a target hot word in the hot word library that matches the text to be identified in the image, and the context information containing the target hot word is determined to obtain the second context information.

[0057] In this embodiment of the disclosure, when acquiring an image to be recognized, the domain of the text to be recognized in the image can be input into the OCR model to obtain a hot word library for that domain. In the obtained hot word library, based on the actual content of the text to be recognized, the K hot words that best match the text are perceived and acquired as target hot words. The hot word perception results are combined with the content of the text to be recognized and decoded; the resulting context information from the hot word decoder is used as second context information.

[0058] In step S103, the first context information and the second context information are fused to obtain fused context information.

[0059] In this embodiment of the disclosure, the first context information and the second context information can be fused according to different weight ratios to obtain fused context information, and the fusion weights can be obtained by training a feature fusion model.

[0060] In step S104, the text recognition result is obtained based on the fused context information.

[0061] In this embodiment of the disclosure, each character in the text to be recognized is output based on the obtained fusion context information to obtain an editable text recognition result.

[0062] In the OCR text recognition method based on hot word awareness provided in this embodiment, a hot word awareness branch is added without destroying the original OCR recognition model. In the decoding stage, a hot word library matching mechanism is introduced to increase the attention of hot words and improve the recognition probability by matching image features and hot word features (visual features + semantic features).

[0063] Figure 2 This is a flowchart illustrating a method for determining target hot words in a hot word library that match text to be identified in an image, according to an exemplary embodiment. Figure 2 As shown, the method includes steps S201 to S202.

[0064] In this embodiment of the disclosure, the hot word library includes hot words, as well as the visual features and semantic features corresponding to the hot words.

[0065] In this embodiment, the hot word library contains hot words, and each hot word corresponds to a set of hot word features. Each set of hot word features includes visual features and semantic features. These two types of features can exist in pairs, meaning that each hot word in the hot word library can have a corresponding pair of visual features and semantic features. The visual features can be obtained by rendering the hot words into fonts and then encoding them using a CNN, or by encoding existing hot word character images. The semantic features can be obtained using BERT encoding.

[0066] In an exemplary embodiment of this disclosure, the hot word library contains at least one hot word and its corresponding hot word features. For example, hot word w has n characters, and its hot word features are: , where the feature sequence For the visual features of the hot word w, f w These are semantic features.

[0067] In step S201, the image feature sequence of the image during the optical character recognition process is obtained.

[0068] In this embodiment of the disclosure, when performing OCR recognition on an image, the image is encoded to obtain an image feature sequence. A CNN encoder or a Vision Transformer (ViT) encoder can be used to encode the image, extracting multi-level visual features from the image. These multi-level visual features constitute the image feature sequence of the image to be recognized. The image feature sequence can be... Let N represent the number of characters in the text to be recognized.

[0069] In step S202, target hot words whose visual features and semantic features match the image feature sequence are determined in the hot word library.

[0070] In this embodiment, the visual features corresponding to each hot word in the hot word library are compared with the image feature sequence obtained by encoding the image, and the image feature sequence is decoded to obtain the context information at time t. Time t can be a relative time in the decoding process. The semantic features of the hot words are compared with the context information of the image feature sequence at time t-1 to determine K hot words whose visual and semantic features both match the image feature sequence; these K hot words are then selected as target hot words. The value of K can be customized.

[0071] Traditional OCR models typically employ general decoding strategies (such as CTC or Attention mechanisms), which are difficult to optimize for specific domain vocabulary, leading to errors or omissions in the recognition of key information.

[0072] The method for determining target hot words that match the text to be identified in an image by introducing a hot word library, encoding the hot words into vectors, and injecting them into a hot word perception decoder, and matching the image feature sequence with the visual and semantic dual features of each hot word, can select target hot words in the hot word library that are more in line with the real content of the text to be identified.

[0073] Figure 3 This is a flowchart illustrating a method for determining target hot words in a hot word library that match visual and semantic features with image feature sequences, according to an exemplary embodiment. Figure 3 As shown, the method includes steps S301 to S303.

[0074] In step S301, the similarity between the image feature sequence and the visual features of each hot word in the hot word library is determined to obtain the visual similarity, and the similarity between the image feature sequence and the semantic features of each hot word in the hot word library is determined to obtain the semantic similarity.

[0075] In this embodiment of the disclosure, the similarity between the image feature sequence and the visual feature sequence is calculated based on the image feature sequence of the text to be identified and the visual feature sequence contained in the visual features of hot words. The image feature sequence is a multi-level visual feature sequence. The similarity between the image feature sequence and the semantic features is calculated based on first contextual information and the semantic features of hot words. The first contextual information obtained by decoding the image feature sequence can be considered as the semantic features of the image feature sequence.

[0076] In an exemplary embodiment of this disclosure, similarity calculation can be performed using the following formula: Visual similarity: Semantic similarity: Where w0 is the number of characters in the hot word w, For the context information at time t-1, during the similarity calculation process, the following is used: It can predict the similarity between the context information output at time t and the hot words.

[0077] In step S302, the visual similarity and semantic similarity are weighted to obtain the similarity between the image feature sequence and each hot word in the hot word library.

[0078] In step S303, hot words whose similarity to each hot word in the hot word library is higher than the similarity threshold are taken as target hot words.

[0079] In this embodiment, visual features and semantic features are fused and calculated according to different weights to obtain the similarity between the image feature sequence and each hot word. The weight ratio can be set in a custom way. The similarity between the image feature sequence and the hot words is sorted to obtain the K hot words with the highest similarity scores, which are used as target hot words.

[0080] In an exemplary embodiment of this disclosure, the similarity weighting can be calculated in the following manner: The weight parameter β can be customized according to requirements.

[0081] The method for calculating similarity provided in this disclosure fuses the visual similarity and semantic similarity of image feature sequences and hot words according to different weight ratios, and can dynamically adjust the influence of visual features and semantic features on the output results.

[0082] Figure 4 This is a flowchart illustrating a method for determining the contextual information of a target hot word and obtaining second contextual information according to an exemplary embodiment, such as... Figure 4 As shown, the method includes steps S401 to S402.

[0083] In step S401, attention weights for image features and characters in the target hot words are obtained based on the image feature sequence, semantic features, and visual features of each character in the target hot words.

[0084] In this embodiment of the disclosure, by introducing an attention mechanism and calculating the attention weight of each character, the OCR model can perform context-dependent modeling in long-sequence text recognition, thereby improving the accuracy of text recognition.

[0085] In this embodiment of the disclosure, the attention weight can be calculated in the following manner: , where V a , w h , w f , w v b is a learnable parameter that can be learned and trained by the model. There are Ks characters in total among the K hot words, f wj , v j Let be the semantic features and visual features of the j-th character, respectively, where 1 ≤ j ≤ Ks.

[0086] In step S402, the attention weights of each character in the target hot word are normalized, and the context information of the target hot word is obtained based on the normalized attention weights, the image feature sequence, the semantic features and visual features of each character in the target hot word, as the second context information.

[0087] In this embodiment of the disclosure, the attention weights of each character of the target hot word in the image feature sequence are normalized and calculated using the softmax function, as shown in the following formula: Based on the normalized attention weight image features h j and the characteristics of the j-th character Calculate the second context information: .

[0088] In this embodiment, an attention mechanism is introduced to enable the OCR model to dynamically focus on the most relevant regions of image features, thereby improving the accuracy of character recognition. An encoder-decoder framework based on an attention mechanism (e.g., Seq2Seq+Attention) allows the decoder to adaptively focus on different spatial positions of the encoder output through attention weights, avoiding alignment problems and improving the recognition performance of complex text (e.g., curved, occluded, or multilingual text).

[0089] Figure 5 This is a flowchart illustrating a method for fusing first context information and second context information to obtain fused context information, according to an exemplary embodiment. Figure 5 As shown, the method includes steps S501 to S502.

[0090] In step S501, the gating value for feature fusion is determined based on the first context information and the second context information.

[0091] In step S502, the fusion ratio of the first context information and the second context information is dynamically adjusted based on the gating value, and the fused context information is obtained based on the adjusted fusion ratio.

[0092] In this embodiment of the disclosure, the sigmoid activation function can be used to calculate the gate value: Among them, W g and b g These are learnable parameters, obtained through model learning and training. σ is the sigmoid activation function. The gating value controls the proportion of the first and second contextual information in the fused contextual information. The fused contextual information is obtained by fusing the first contextual information recognized by OCR and the second contextual information perceived by hot words using the gating value in the following calculation method: It can dynamically adjust the impact of the hot word perception module's results on the OCR recognition results, improving the accuracy of hot word recognition while ensuring the accuracy of ordinary character recognition.

[0093] Figure 6 This is a flowchart illustrating a method for obtaining text recognition results based on fused contextual information, according to an exemplary embodiment. Figure 6 As shown, the method includes step S601.

[0094] In step S601, the bias probability of each character in the text to be recognized is determined, and the text recognition result is obtained based on the bias probability and the fused context information.

[0095] In this embodiment of the disclosure, the hidden state can be calculated using a gating unit: ,in, The embedding vector is defined for the final output at time t-1, and the output bias probability of each character can be calculated as follows: The final text recognition result is output based on the above steps, making the recognition result more accurate.

[0096] Figure 7 This is a flowchart illustrating a method for obtaining a hot word database according to an exemplary embodiment, such as... Figure 7 As shown, the method includes steps S701 to S702.

[0097] In step S701, in response to obtaining the domain of the text to be identified, the hot word library matching the domain is used as the target hot word library.

[0098] In step S702, in response to the absence of a domain for which the text to be identified is not obtained, the hot word library of all domains is used as the target hot word library.

[0099] In this embodiment of the disclosure, when inputting an image containing text to be recognized, a relevant field (e.g., finance, healthcare) can be input, and a relevant hot word library (e.g., finance hot word library, healthcare hot word library, etc.) can be selected from the entire hot word library as the target hot word library. If no relevant field for the text to be recognized is input, then during the text recognition process, the entire hot word library is used as the target hot word library. The target hot word library can be represented as: That is, the target hot word library contains M hot words.

[0100] In this embodiment of the disclosure, a custom hot word library can be imported before image recognition. The custom hot word library can be used together with the input relevant domain hot word library as the target hot word library. Customizing the hot word library according to the business scenario can improve cross-scenario generalization performance.

[0101] In this embodiment of the disclosure, the hot word library can be obtained through collection and organization. In one example, some general fields can obtain real-time traffic hot words from platforms such as social media or search engines, and organize these hot words to obtain a hot word library for the relevant field. In another example, vertical industries can obtain hot words from different industry fields from platforms such as industry vertical platforms and industry reports / databases, and organize these hot words to obtain a hot word library for the relevant field. In yet another example, academic and scientific research fields can obtain academic hot words for the relevant field from Chinese and English academic platforms.

[0102] In this embodiment of the disclosure, the hot word library can also be customized according to needs. For example, when a document needs to be processed separately, a dedicated hot word library can be set up for that document. This dedicated hot word library can be applied only to this document or it can be incorporated into the entire hot word library.

[0103] Figure 8 This is a flowchart illustrating an OCR text recognition method based on hot word awareness, according to an exemplary embodiment.

[0104] like Figure 8 As shown, after inputting an image containing the text to be recognized, OCR recognition is performed on the image. This involves encoding the image to obtain an image feature sequence, then decoding the image feature sequence to obtain the OCR recognition result. Without disrupting the original OCR recognition process, a hot word awareness module is added, including a hot word database, a hot word retrieval module, and a hot word attention calculation module. Based on the results from the OCR recognition module and the hot word awareness module, feature fusion is performed through gating values. The bias output probability of each character is calculated based on the fused features (fused context information), and the text recognition result is output according to the bias probability of each character.

[0105] The OCR text recognition method based on hot word perception provided in this embodiment introduces a hot word library to achieve hot word prior embedding. Hot words are encoded into vectors and input into the hot word perception module. Combining the OCR recognition results and the hot word perception results, a new hot word perception module is added without destroying the original OCR model. This does not destroy the main structure of the original OCR recognition, does not require retraining of the entire model, and can dynamically adjust the impact of the hot word perception module results on the final recognition effect for text recognition. While ensuring the accuracy of OCR recognition of text images, it improves the accuracy of the model in perceiving hot words in different fields.

[0106] The OCR text recognition method based on hot word perception provided in this disclosure reduces misidentification of easily confused characters and improves the accuracy of hot word perception in specific scenarios by adding a hot word perception module. It achieves "controllable" recognition by introducing a hot word library to meet personalized needs, such as medical hot word libraries or financial hot word libraries. Furthermore, the hot word library can be customized according to business scenarios to improve cross-scenario generalization performance. Based on the matching between the image features of the image to be recognized and the hot word features, the impact of the hot word perception module's results on the final recognition effect can be dynamically adjusted, improving the accuracy of hot word recognition while maintaining the accuracy of ordinary OCR recognition.

[0107] This disclosed embodiment improves the OCR recognition's ability to detect hot words and the accuracy of the recognition results by adding a hot word perception module to the OCR recognition module, thus addressing the shortcomings of traditional OCR recognition.

[0108] 1. Lack of sensitivity to hot words: Traditional models treat hot words and non-hot words equally, and cannot use prior knowledge to improve the accuracy of hot word perception.

[0109] 2. Poor domain adaptability: It lacks the ability to perceive domain-specific vocabulary, and the accuracy drops when transferred to specific business scenarios. The entire model needs to be retrained for different domains (legal / medical).

[0110] 3. Static dictionary, unable to dynamically update the hot word database.

[0111] The OCR text recognition method based on hot word perception provided in this disclosure can improve the ability of traditional OCR recognition methods to perceive hot words and improve the accuracy of OCR recognition.

[0112] Based on the same concept, this disclosure also provides an OCR text recognition device 100 based on hot word perception.

[0113] It is understood that the OCR text recognition device 100 based on hot word perception provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0114] Figure 9 This is a block diagram illustrating a hot word-aware OCR text recognition device 100 according to an exemplary embodiment. (Refer to...) Figure 9 The device includes an acquisition unit 101 and a processing unit 102.

[0115] The acquisition unit 101 is used to acquire an image including the text to be identified and to acquire a hot word library, which includes hot words.

[0116] The processing unit 102 is used to perform optical character recognition on the image to obtain first context information and perceive target hot words. The perceived target hot words are determined by identifying target hot words in the hot word library that match the text to be recognized in the image, and determining the context information containing the target hot words to obtain second context information. The first context information and the second context information are then fused to obtain fused context information. Based on the fused context information, the text recognition result is obtained.

[0117] In some implementations, the hot word library includes hot words, as well as the visual features and semantic features corresponding to the hot words. The processing unit 102 perceives the target hot words in the following way: the target hot words are determined in the hot word library to match the target hot words that are to be recognized in the image. The image feature sequence of the image is obtained in the process of optical character recognition, and the target hot words whose visual features and semantic features match the image feature sequence are determined in the hot word library.

[0118] In some implementations, the processing unit 102 determines target hot words in the hot word library whose visual features and semantic features match the image feature sequence in the following manner: determining the similarity between the visual features of the image feature sequence and each hot word in the hot word library to obtain visual similarity, and determining the similarity between the semantic features of the image feature sequence and each hot word in the hot word library to obtain semantic similarity, weighting the visual similarity and semantic similarity to obtain the similarity between the image feature sequence and each hot word in the hot word library, and taking the hot words whose similarity between the image feature sequence and each hot word in the hot word library is higher than the similarity threshold as target hot words.

[0119] In some implementations, the processing unit 102 determines the context information of the target hot word in the following manner to obtain the second context information: based on the image feature sequence, the semantic features and visual features of each character in the target hot word, the attention weights of the image features and each character in the target hot word are obtained, the attention weights of the image features and each character in the target hot word are normalized, and the context information of the target hot word is obtained based on the normalized attention weights, the image feature sequence, the semantic features and visual features of each character in the target hot word, as the second context information.

[0120] In some implementations, the processing unit 102 fuses the first context information and the second context information in the following manner to obtain fused context information: based on the first context information and the second context information, a threshold value for feature fusion is determined; based on the threshold value, the fusion ratio of the first context information and the second context information is dynamically adjusted; and the fused context information is obtained based on the adjusted fusion ratio.

[0121] In some implementations, the processing unit 102 obtains the text recognition result based on the fusion context information by determining the bias probability of each character in the text to be recognized, and obtaining the text recognition result based on the bias probability and the fusion context information.

[0122] In some implementations, the hot word library includes hot word libraries for multiple fields. The acquisition unit 101 acquires the hot word library in the following manner: in response to acquiring the field of the text to be identified, the hot word library matching the field is used as the target hot word library; or in response to not acquiring the field of the text to be identified, the hot word libraries of all fields in the multiple fields are used as the target hot word library.

[0123] This disclosure also provides an electronic device, including a processor; a memory for storing processor-executable computer programs or instructions; wherein the processor is configured to execute the computer programs or instructions to implement the steps of any of the hot word-aware OCR text recognition methods described above.

[0124] This disclosure also provides a storage medium storing a computer program or instructions, which, when executed by a processor of an electronic device, enables the electronic device to perform the steps of the hot word-aware OCR text recognition method as described in any of the above embodiments.

[0125] Figure 10 This is a block diagram illustrating an apparatus 200 for hotword-based OCR text recognition according to an exemplary embodiment. The apparatus 200 can be provided as a terminal. For example, the apparatus 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0126] Reference Figure 10 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0127] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0128] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0129] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0130] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0131] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0132] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0133] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0134] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0135] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0136] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0137] Figure 11 This is a block diagram illustrating an apparatus 300 for OCR text recognition based on hotword awareness, according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (Refer to...) Figure 11 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the aforementioned text recognition method.

[0138] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0139] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 332 including instructions, which can be executed by the processing component 322 of the device 300 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0140] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0141] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the aforementioned hot word-aware OCR text recognition method.

[0142] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, a first component, part, region, layer, or section mentioned in the examples may also be referred to as a second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature.

[0143] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0144] In this description, "multiple" means at least two, referring to two or more, such as two, three, etc., unless otherwise explicitly specified. Other quantifiers are similar. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, unless otherwise specified or clearly indicated from the context, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more."

[0145] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of two or more; "and / or" describes the association relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Similarly, "at least one of..." includes any one of the related listed items and any combination of two or more.

[0146] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0147] Furthermore, the term "exemplary" is used herein to indicate that it serves as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term "exemplary" is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to indicate an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to indicate any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then applying A or B satisfies the condition under any of the foregoing instances.

[0148] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in this disclosure, such terms are intended to be inclusive in a manner similar to the term “including.”

[0149] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0150] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for OCR text recognition based on hot word awareness, characterized in that, include: Acquire an image containing the text to be recognized, perform optical character recognition on the image, and obtain first context information; A hot word library is acquired, a target hot word is perceived, and context information containing the target hot word is determined to obtain second context information. The hot word library includes hot words, as well as visual features and semantic features corresponding to the hot words. The perception of the target hot word is to determine the target hot word in the hot word library that matches the text to be identified in the image. The first context information and the second context information are fused to obtain fused context information; Based on the fused context information, the text recognition result is obtained; Among them, the target hot words for perception include: Obtain the image feature sequence of the image during the optical character recognition process; In the hot word library, target hot words whose visual and semantic features match the image feature sequence are identified; Specifically, determining the context information of the target hot words to obtain second context information includes: Based on the image feature sequence, the semantic features and visual features of each character in the target hot word, the attention weights of the image features and each character in the target hot word are obtained; The attention weights of each character in the image features and the target hot words are normalized, and the context information of the target hot words is obtained based on the normalized attention weights, the image feature sequence, the semantic features and visual features of each character in the target hot words, as the second context information.

2. The method according to claim 1, characterized in that, The step of determining target hot words in the hot word library that match the image feature sequence in terms of visual and semantic features includes: The similarity between the image feature sequence and the visual features of each hot word in the hot word library is determined to obtain the visual similarity, and the similarity between the image feature sequence and the semantic features of each hot word in the hot word library is determined to obtain the semantic similarity. By weighting the visual similarity and the semantic similarity, the similarity between the image feature sequence and each hot word in the hot word library is obtained; The hot words whose similarity to each hot word in the hot word library is higher than the similarity threshold of the image feature sequence are taken as target hot words.

3. The method according to claim 1, characterized in that, The process of fusing the first context information and the second context information to obtain fused context information includes: Based on the first context information and the second context information, a feature fusion gate value is determined; Based on the gating value, the fusion ratio of the first context information and the second context information is dynamically adjusted, and the fused context information is obtained based on the adjusted fusion ratio.

4. The method according to claim 1, characterized in that, The process of obtaining text recognition results based on the fused context information includes: The bias probability of each character in the text to be recognized is determined, and the text recognition result is obtained based on the bias probability and the fusion context information.

5. The method according to claim 1, characterized in that, The hot word library includes hot word libraries from multiple fields, and the process of obtaining the hot word library includes: In response to obtaining the domain of the text to be identified, the hot word library matching the domain is used as the target hot word library; or In response to a domain where the text to be identified is not obtained, the hot word library of all domains in the multiple domains is used as the target hot word library.

6. A text recognition device based on hot word perception, characterized in that, include: The acquisition unit is used to acquire an image including the text to be identified and to acquire a hot word library, which includes hot words and the visual and semantic features corresponding to the hot words; The processing unit is configured to perform optical character recognition on the image to obtain first context information and perceived target hot words. The perceived target hot words are determined in the hot word library to match the text to be recognized in the image, and context information containing the target hot words is determined to obtain second context information. The first context information and the second context information are then fused to obtain fused context information. Based on the fused context information, a text recognition result is obtained. The processing unit perceives target hot words in the following manner: Obtain the image feature sequence of the image during the optical character recognition process; In the hot word library, target hot words whose visual and semantic features match the image feature sequence are identified; The processing unit determines the context information of the target hot word in the following manner to obtain the second context information: Based on the image feature sequence, the semantic features and visual features of each character in the target hot word, the attention weights of the image features and each character in the target hot word are obtained; The attention weights of each character in the image features and the target hot words are normalized, and the context information of the target hot words is obtained based on the normalized attention weights, the image feature sequence, the semantic features and visual features of each character in the target hot words, as the second context information.

7. The apparatus according to claim 6, characterized in that, The processing unit determines target hot words in the hot word library whose visual and semantic features match the image feature sequence in the following manner: The similarity between the image feature sequence and the visual features of each hot word in the hot word library is determined to obtain the visual similarity, and the similarity between the image feature sequence and the semantic features of each hot word in the hot word library is determined to obtain the semantic similarity. By weighting the visual similarity and the semantic similarity, the similarity between the image feature sequence and each hot word in the hot word library is obtained; The hot words whose similarity to each hot word in the hot word library is higher than the similarity threshold of the image feature sequence are taken as target hot words.

8. The apparatus according to claim 6, characterized in that, The processing unit fuses the first context information and the second context information in the following manner to obtain fused context information: Based on the first context information and the second context information, a feature fusion gate value is determined; Based on the gating value, the fusion ratio of the first context information and the second context information is dynamically adjusted, and the fused context information is obtained based on the adjusted fusion ratio.

9. The apparatus according to claim 6, characterized in that, The processing unit obtains the text recognition result based on the fused context information in the following manner: The bias probability of each character in the text to be recognized is determined, and the text recognition result is obtained based on the bias probability and the fusion context information.

10. The apparatus according to claim 6, characterized in that, The hot word library includes hot word libraries from multiple fields. The acquisition unit acquires the hot word library in the following manner: In response to obtaining the domain of the text to be identified, the hot word library matching the domain is used as the target hot word library; or In response to a domain where the text to be identified is not obtained, the hot word library of all domains in the multiple domains is used as the target hot word library.

11. An electronic device, characterized in that, include: processor: Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the OCR text recognition method based on hot word perception as described in any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by the processor of an electronic device, enable the electronic device to perform the OCR text recognition method based on hot word perception as described in any one of claims 1 to 5.

Images