Text recognition method, device and equipment

Abstract

This application discloses a text recognition method, apparatus, electronic device, and storage medium, comprising: acquiring character encoding intervals for at least one preset language and constructing a tree structure based on the character encoding intervals; acquiring the text to be processed in real time; for each character in the text to be processed, traversing and querying the tree structure to determine the character encoding interval to which the character belongs, and taking the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character; classifying the characters according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language. This achieves real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multilingual recognition needs in big data real-time processing scenarios and improve the processing performance and application effect of text recognition.

Description

Technical Field

[0001] This application belongs to the field of data processing, and specifically relates to a text recognition method, apparatus, device, and storage medium. Background Technology

[0002] With increasing language exchanges between different countries and regions, the need for real-time text recognition in less commonly spoken languages ​​is becoming more urgent. For example, in applications such as social media monitoring, real-time translation, online customer service, and information security, the rapid and accurate identification and processing of text in less commonly spoken languages ​​is crucial for timely acquisition of key information and rapid response.

[0003] In existing technologies, methods for recognizing text in minority languages ​​mainly rely on offline batch processing, which requires the prior collection of a large amount of text data in minority languages, and then the processing of this text data in batches.

[0004] However, this method is only applicable to processing static minority language text data that does not change or update frequently. When processing real-time minority language text data that is constantly generated and needs to be processed immediately, it cannot immediately process new minority language text data and provide results, resulting in processing delays. In other words, the traditional offline batch processing mode has insufficient real-time processing capabilities and cannot respond to minority language recognition needs in a timely manner, resulting in poor timeliness and application effects of minority language text recognition. Summary of the Invention

[0005] The purpose of this application is to provide a text recognition method, apparatus, device, and storage medium that can solve the problem that current methods for recognizing text in less commonly spoken languages ​​lack real-time processing capabilities, cannot respond to the needs of recognizing text in less commonly spoken languages ​​in a timely manner, and thus result in poor timeliness and application effect of text recognition in less commonly spoken languages.

[0006] In a first aspect, embodiments of this application provide a text recognition method, the method comprising:

[0007] Obtain the character encoding range of at least one preset language, and construct a tree structure based on the character encoding range;

[0008] Real-time acquisition of text to be processed;

[0009] For each character in the text to be processed, the tree structure is traversed and queried to determine the character encoding range to which the character belongs, and the preset language corresponding to the character encoding range to which the character belongs is taken as the target language corresponding to the character.

[0010] Based on the target language corresponding to each character, the characters are classified to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language.

[0011] Optionally, obtaining the character encoding range of at least one preset language includes:

[0012] Establish a connection with a remote dictionary service;

[0013] Read the character encoding range of at least one preset language from the remote dictionary service.

[0014] Optionally, constructing the tree structure based on the character encoding range includes:

[0015] Create the first node of the tree structure, which serves as the root node;

[0016] Select a target interval from the character encoding interval of at least one of the preset languages, and use it as the node interval corresponding to the root node;

[0017] The character encoding intervals other than the target interval are traversed sequentially as candidate intervals. Based on the interval relationship between the candidate intervals and the node intervals corresponding to the tree structure, the position of the node corresponding to the candidate interval in the tree structure is determined, and a new node is created at the determined position. The candidate interval is used as the node interval corresponding to the new node to obtain the tree structure.

[0018] Optionally, after constructing the tree structure based on the character encoding range, the method further includes:

[0019] The tree structure is stored in memory;

[0020] The real-time acquisition of the text to be processed includes:

[0021] The text to be processed is acquired in real time and stored in the memory.

[0022] The step of traversing and querying the tree structure to determine the character encoding range to which the character belongs includes:

[0023] In the memory, the tree structure is traversed and queried to determine the character encoding range to which the character belongs.

[0024] Optionally, the real-time acquisition of the text to be processed includes:

[0025] Obtain the input data and the configuration information of the input data;

[0026] Based on the configuration information, the text to be processed is extracted from the input data.

[0027] Optionally, after obtaining the input data, the process includes:

[0028] Perform a data validity check on the input data;

[0029] If the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0030] Optionally, after obtaining the input data, the process includes:

[0031] Obtain the running label of the text recognition operator;

[0032] When the running label indicates that the text recognition operator has started running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0033] Optionally, the text to be processed includes at least one dataset; after acquiring the text to be processed in real time, the process includes:

[0034] Get the whitelist;

[0035] Detect whether the dataset is in the whitelist, and remove datasets not in the whitelist from the text to be processed.

[0036] Optionally, after acquiring the text to be processed in real time, the method further includes:

[0037] If the text to be processed is unstructured text, the text to be processed is deduplicated to obtain optimized text.

[0038] Optionally, the step of traversing and querying the tree structure for each character in the text to be processed to determine the character encoding range to which the character belongs includes:

[0039] Each character in the text to be processed is taken as the target character in turn, and the character encoding range of the previous character of the target character is obtained as the cache range;

[0040] Determine whether the target character belongs to the cache range;

[0041] If the target character belongs to the cache range, then it is determined that the target character is the same as the character encoding range corresponding to the previous character of the target character;

[0042] If the target character does not belong to the cache range, the tree structure is traversed to determine the character encoding range to which the target character belongs.

[0043] Optionally, obtaining the character encoding range of the previous character of the target character as a cache range includes:

[0044] Retrieve cached key-value pairs; the key of the cached key-value pair is the previous character of the target character, and the value of the cached key-value pair is the cache range;

[0045] After determining that the target character is the same as the character encoding range corresponding to the previous character of the target character, the method further includes:

[0046] Update the key of the cached key-value pair based on the target character;

[0047] After traversing the tree structure to determine the character encoding range to which the target character belongs, the method further includes:

[0048] Update the key of the cached key-value pair based on the target character;

[0049] Update the value of the cached key-value pair based on the character encoding range to which the target character belongs.

[0050] Secondly, embodiments of this application provide a text recognition device, the device comprising:

[0051] A construction module is used to obtain the character encoding range of at least one preset language and construct a tree structure based on the character encoding range;

[0052] The acquisition module is used to acquire the text to be processed in real time;

[0053] The traversal module is used to traverse and query the tree structure for each character in the text to be processed, determine the character encoding interval to which the character belongs, and take the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character.

[0054] The output module is used to classify the characters according to the target language corresponding to each character, and obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language.

[0055] Optionally, the construction module is used for:

[0056] Establish a connection with a remote dictionary service;

[0057] Read the character encoding range of at least one preset language from the remote dictionary service.

[0058] Optionally, the construction module is used for:

[0059] Create the first node of the tree structure, which serves as the root node;

[0060] Select a target interval from the character encoding interval of at least one of the preset languages, and use it as the node interval corresponding to the root node;

[0061] The character encoding intervals other than the target interval are traversed sequentially as candidate intervals. Based on the interval relationship between the candidate intervals and the node intervals corresponding to the tree structure, the position of the node corresponding to the candidate interval in the tree structure is determined, and a new node is created at the determined position. The candidate interval is used as the node interval corresponding to the new node to obtain the tree structure.

[0062] Optionally, the construction module is further configured to:

[0063] The tree structure is stored in memory;

[0064] The acquisition module is also used for:

[0065] The text to be processed is acquired in real time and stored in the memory.

[0066] The traversal module is specifically used for:

[0067] In the memory, the tree structure is traversed and queried to determine the character encoding range to which the character belongs.

[0068] Optionally, the acquisition module is used to:

[0069] Obtain the input data and the configuration information of the input data;

[0070] Based on the configuration information, the text to be processed is extracted from the input data.

[0071] Optionally, the acquisition module is used to:

[0072] Perform a data validity check on the input data;

[0073] If the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0074] Optionally, the acquisition module is used to:

[0075] Obtain the running label of the text recognition operator;

[0076] When the running label indicates that the text recognition operator has started running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0077] Optionally, the text to be processed includes at least one dataset; the acquisition module is configured to:

[0078] Get the whitelist;

[0079] Detect whether the dataset is in the whitelist, and remove datasets not in the whitelist from the text to be processed.

[0080] Optionally, the acquisition module is used to:

[0081] If the text to be processed is unstructured text, the text to be processed is deduplicated to obtain optimized text.

[0082] Optionally, the traversal module is used for:

[0083] Each character in the text to be processed is taken as the target character in turn, and the character encoding range of the previous character of the target character is obtained as the cache range;

[0084] Determine whether the target character belongs to the cache range;

[0085] If the target character belongs to the cache range, then it is determined that the target character is the same as the character encoding range corresponding to the previous character of the target character;

[0086] If the target character does not belong to the cache range, the tree structure is traversed to determine the character encoding range to which the target character belongs.

[0087] Optionally, the traversal module is used for:

[0088] Retrieve cached key-value pairs; the key of the cached key-value pair is the previous character of the target character, and the value of the cached key-value pair is the cache range;

[0089] After determining that the target character is the same as the character encoding range corresponding to the previous character of the target character, the method further includes:

[0090] Update the key of the cached key-value pair based on the target character;

[0091] After traversing the tree structure to determine the character encoding range to which the target character belongs, the method further includes:

[0092] Update the key of the cached key-value pair based on the target character;

[0093] Update the value of the cached key-value pair based on the character encoding range to which the target character belongs.

[0094] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0095] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0096] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0097] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0098] In the solution provided in this application embodiment, at least one preset language character encoding range is obtained, and a tree structure is constructed based on the character encoding range; the text to be processed is obtained in real time; for each character in the text to be processed, the query tree structure is traversed to determine the character encoding range to which the character belongs, and the preset language corresponding to the character encoding range to which the character belongs is taken as the target language corresponding to the character; the characters are classified according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language.

[0099] As can be seen from the above, this application proposes a text recognition method that constructs a tree structure using the character encoding range of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text can be identified by traversing and querying the tree structure to obtain the recognition result, thereby realizing real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multilingual recognition needs in big data real-time processing scenarios and improve the processing performance and application effect of text recognition. Attached Figure Description

[0100] Figure 1 This is a flowchart illustrating a text recognition method according to an exemplary embodiment;

[0101] Figure 2 This is a logical schematic diagram illustrating a method for identifying minority languages ​​according to an exemplary embodiment;

[0102] Figure 3This is a block diagram illustrating a text recognition device according to an exemplary embodiment;

[0103] Figure 4 This is a block diagram illustrating an electronic device according to an exemplary embodiment;

[0104] Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0105] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0106] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0107] The text recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0108] Figure 1 This is a flowchart illustrating a text recognition method according to an exemplary embodiment, the text recognition method comprising the following steps.

[0109] In step S11, the character encoding range of at least one preset language is obtained, and a tree structure is constructed based on the character encoding range.

[0110] With increasing language exchanges between different countries and regions, the need for real-time text recognition in less commonly spoken languages ​​is becoming more urgent. Related technologies primarily rely on offline batch processing methods for text recognition in less commonly spoken languages, requiring the pre-collection of large amounts of text data in these languages ​​before processing it in batches.

[0111] However, this method is only applicable to static minority language text data processing that does not change or update frequently. When processing real-time minority language text data that is constantly generated and needs to be processed immediately, it cannot immediately process new minority language text data and provide results. There is a processing delay, and it cannot respond to minority language recognition needs in a timely manner, resulting in poor timeliness and application effect of minority language text recognition.

[0112] Based on this, this application proposes a text recognition method to solve the above problems, meet the multilingual recognition needs in big data real-time processing scenarios, and improve the processing performance and application effect of text recognition.

[0113] In this step, the first step is to obtain the character encoding range (Unicode) of one or more preset languages. The character encoding range is designed to set a unified and unique binary code for each character in each language to meet the needs of cross-language and cross-platform text conversion and processing. In other words, each character included in each preset language will uniquely correspond to a part of the character encoding range. This part of the character encoding range may be continuous or divided into several segments, and there is no specific limitation.

[0114] In one implementation, obtaining the character encoding range of at least one preset language includes:

[0115] Establish a connection with a remote dictionary service;

[0116] Read the character encoding range of at least one preset language from a remote dictionary service.

[0117] Among them, the Remote Dictionary Server (Redis) is used to store and manage character encoding ranges for various preset languages. It has efficient data access capabilities, supports dynamic updates and fast queries, to ensure the accuracy and timeliness of text recognition.

[0118] In other words, by setting up and initializing the Redis connection, you can read the character encoding ranges of various preset languages ​​from Redis for subsequent character queries and recognition.

[0119] After obtaining the character encoding range of the preset language, a tree structure can be constructed based on the character encoding range to facilitate subsequent range query operations based on the tree structure. The tree structure is a non-linear data structure consisting of a hierarchical set of n (n≥0) finite nodes, where each node contains a data element and several branches pointing to its subtree.

[0120] One implementation involves constructing a tree structure based on character encoding ranges, including:

[0121] Create the first node of the tree structure, which serves as the root node;

[0122] Select the target interval from the character encoding interval of at least one preset language, and use it as the node interval corresponding to the root node;

[0123] Iterate through all character encoding intervals except the target interval, and use them as candidate intervals. Based on the interval relationship between the candidate intervals and the intervals of each node in the tree structure, determine the position of the node corresponding to the candidate interval in the tree structure, and create a new node at the determined position. Use the candidate interval as the node interval corresponding to the new node to obtain the tree structure.

[0124] In other words, the read character encoding ranges can be constructed into a tree structure to enable efficient queries. First, a tree structure needs to be defined to store the character encoding ranges for each preset language. Each node in the tree structure represents a character encoding range for a preset language. Then, each character encoding range read from Redis needs to be inserted into the tree structure.

[0125] Various tree structures can be selected to store character encoding ranges, such as binary search trees (BSTs), balanced binary search trees (e.g., AVL trees, red-black trees), B-trees, or B+ trees. Each tree structure has its specific advantages and applicable scenarios.

[0126] For example, consider a binary search tree. Each node contains a character encoding range (defined by the start and end characters) and pointers to the left and right subtrees. Suppose the following character encoding ranges need to be inserted into the tree structure:

[0127] Interval 1: [U+0041, U+005A] (uppercase letters A to Z)

[0128] Interval 2: [U+0061, U+007A] (lowercase letters a to z)

[0129] Interval 3: [U+03A0, U+03C1] (Greek letters Π to Ρ)

[0130] First, create a new node with the starting character U+0041 and the ending character U+005A. Since the tree is empty, this node becomes the root node.

[0131] Next, create a new node with the starting character U+0061 and the ending character U+007A. Compare the starting character U+0061 of the new node with the starting character U+0041 of the root node. Since U+0061 is greater than U+0041, recursively insert the node into the right subtree of the new node. At this point, the root node has no right subtree, so the new node becomes the right child of the root node.

[0132] Next, create a new node with the starting character U+03A0 and the ending character U+03C1. Compare the starting character U+03A0 of the new node with the starting character U+0041 of the root node. Since U+03A0 is greater than U+0041, recursively continue the comparison at the right subtree position of the new node. Compare U+03A0 with the starting character U+0061 of the right child node (interval 2).

[0133] Since U+03A0 is greater than U+0061, but interval 3 and interval 2 do not overlap, the new node should become the right sibling of the node in interval 2. Insert the new node into the right subtree position of the node in interval 2 (since the node in interval 2 currently has no right subtree, the new node directly becomes its right child).

[0134] The final tree structure is as follows:

[0135] Root node: [U+0041,U+005A] (uppercase letters A to Z)

[0136] Right child node: [U+0061, U+007A] (lowercase letters a to z)

[0137] Right child node: [U+03A0, U+03C1] (Greek letters Π to Ρ)

[0138] It is understandable that tree structures are naturally suitable for representing hierarchical relationships, making it easy to manage and query the hierarchical relationships of character encoding ranges; moreover, tree structures allow query operations on character encoding ranges to be completed in logarithmic time complexity, thereby improving query efficiency; in addition, tree structures make it easy to insert, delete, and update character ranges, adapting to the needs of dynamic changes in character encoding ranges.

[0139] In step S12, the text to be processed is acquired in real time.

[0140] In this step, the text to be processed can be acquired in real time. The text to be processed is the text that needs to be recognized. The data source of the text to be processed may include online systems, real-time databases, data streams, etc., and there are no specific limitations.

[0141] In this application, the text recognition operator can be implemented in the `processElement` method of `ProcessFunction` in Flink. By leveraging Flink's stream processing framework, millisecond-level processing of real-time acquired text can be achieved, enabling rapid response to real-time text recognition needs and ensuring the timeliness and accuracy of data processing.

[0142] In one implementation, the text to be processed is acquired in real time, including:

[0143] Obtain input data and its configuration information;

[0144] Based on the configuration information, extract the text to be processed from the input data.

[0145] In other words, after obtaining the input data, it is necessary to further determine how to obtain the text to be processed based on the configuration information of the input data. For example, it may involve extracting specific fields from the input data, selecting specific dataset codes, or selecting processing methods based on other conditions, etc., without being limited to specifics.

[0146] In one implementation, after obtaining the input data, the process includes:

[0147] Perform data validity checks on the input data;

[0148] If the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is executed.

[0149] In other words, after obtaining the input data, a data validity check is required. This is to ensure that the input data for subsequent processing is valid and conforms to the expected format. If the data is invalid, such as having a format error, missing key fields, or containing illegal characters, it is returned directly without further processing to save system resources.

[0150] In one implementation, after obtaining the input data, the process includes:

[0151] Obtain the running label of the text recognition operator;

[0152] When the running label indicates that the text recognition operator has started running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0153] In other words, after acquiring the input data, it is necessary to further check whether the text recognition operator needs to be run. This step is a conditional judgment used to control whether the text recognition operator is executed. The decision of whether the text recognition operator needs to run typically depends on the specific application scenario and requirements.

[0154] Specifically, whether the text recognition operator is started or run can be controlled by a run label. If the run label is false, it means that minority language recognition is not needed at present, so the input data is directly output and returned. If the run label is true, it means that minority language recognition is needed at present, so subsequent steps can continue.

[0155] In one implementation, the text to be processed includes at least one dataset; after acquiring the text to be processed in real time, the process includes:

[0156] Get the whitelist;

[0157] Check if the dataset is in the whitelist, and remove datasets not in the whitelist from the text to be processed.

[0158] In other words, after obtaining the text to be processed, a whitelist can be obtained. The whitelist is a predefined collection containing datasets that need to be used for text recognition. Then, it can be checked whether the dataset included in the text to be processed is in the whitelist. If the current dataset is in the whitelist, then text recognition processing continues; otherwise, processing may be skipped or other operations may be performed.

[0159] In one implementation, after acquiring the text to be processed in real time, the following steps are also included:

[0160] When the text to be processed is unstructured, deduplication optimization is performed on the text to be processed to obtain the optimized text.

[0161] Specifically, the steps for deduplication optimization of the text to be processed may include, but are not limited to:

[0162] Text preprocessing: This involves preprocessing the text to be processed, removing duplicate characters and words. By deduplicating the text, redundancy can be reduced, thus lowering computational complexity.

[0163] Efficient data structure: Use efficient data structures (such as Set collections) to store the deduplicated text to be processed, ensuring the uniqueness of the text to be processed and the access speed.

[0164] In this application, configuration parameters for the text recognition operator are set using a parameter tool (such as ParameterTool) to allow for adjustments to the operator's behavior at runtime as needed. The configuration of the text recognition operator is primarily performed in the "open" method of the Flink rich function, ensuring that the text recognition operator runs correctly in subsequent processing.

[0165] For example, the configuration parameters of a text recognition operator may include, but are not limited to:

[0166] The field list parameter specifies which fields in the text to be processed need to be recognized. For example, if the text to be processed is a table containing multiple columns of information, this parameter tells the operator which columns' text may need to be recognized.

[0167] The field list parameter for saving recognition results: After text recognition is completed, it is usually necessary to save the recognition results back to a certain field. This parameter is used to specify which fields will be used to store the recognition results. For example, for each field that is recognized as a certain minor language, there can be a corresponding "Result" field to store the specific minor language information.

[0168] Whitelist Switch Parameter: This is a switch parameter used to enable or disable the text recognition whitelist function. If the whitelist is enabled, only datasets or fields listed in the whitelist will be processed for text recognition. This can be used to optimize performance or to specify that only specific datasets need to be processed.

[0169] The whitelist field list parameter specifies which fields (or datasets) are included in the whitelist when the whitelist switch is enabled. Only these fields of the text to be processed will be used for language recognition. This can be used to ensure that only specific datasets are processed, without having to iterate through the entire text to be processed.

[0170] In step S13, for each character in the text to be processed, the query tree structure is traversed to determine the character encoding range to which the character belongs, and the preset language corresponding to the character encoding range to which the character belongs is taken as the target language corresponding to the character.

[0171] In this step, for each character in the text to be processed, it is necessary to traverse the query tree structure, determine the character encoding range to which the character belongs based on the interval relationship between each node in the tree structure and the character encoding range corresponding to each node, and then take the preset language corresponding to the character encoding range to which the character belongs as the target language of the character.

[0172] In one implementation, for each character in the text to be processed, the query tree structure is traversed to determine the character encoding range to which the character belongs, including:

[0173] Each character in the text to be processed is taken as the target character in turn, and the character encoding range of the previous character of the target character is obtained as the cache range;

[0174] Determine if the target character belongs to the cache range;

[0175] If the target character belongs to the cache range, then it is determined that the target character and the character encoding range corresponding to the previous character are the same.

[0176] If the target character does not belong to the cache range, the query tree structure is traversed to determine the character encoding range to which the target character belongs.

[0177] In other words, when processing consecutive characters, it is assumed that the next character and the previous character belong to the same target language. Therefore, the system first checks whether the cache range of the previous character exists in the cache. If it does, it first searches the cache to see if the target character belongs to the same cache range as the previous character.

[0178] If the target character is found in the cache (i.e., it belongs to the cache range), the result is returned directly, avoiding re-traversing and recognizing the tree structure, thus significantly improving recognition speed. If the target character is not found in the cache (i.e., it does not belong to the cache range), recognition is performed again from the tree structure.

[0179] In one implementation, the character encoding range of the previous character of the target character is obtained and used as a cache range, including:

[0180] Retrieve cached key-value pairs; the key of the cached key-value pair is the character preceding the target character, and the value of the cached key-value pair is the cache range;

[0181] After determining that the target character and the character encoding range corresponding to the previous character of the target character are the same, the following steps are also included:

[0182] Update the key of the cached key-value pair based on the target character;

[0183] After traversing the query tree structure to determine the character encoding range to which the target character belongs, the process also includes:

[0184] Update the key of the cached key-value pair based on the target character;

[0185] Update the value of the cached key-value pair based on the character encoding range to which the target character belongs.

[0186] In other words, to improve the efficiency of consecutive character recognition, the Caffeine caching mechanism can be used. Caffeine is a high-performance Java caching library that can effectively improve cache hit rate and reduce redundant calculations. Specifically, a cache structure can be designed using the Caffeine caching library to store the previous character and its corresponding character encoding range. This cache structure can be represented as a key-value pair, where the key is the previously recognized character and the value is the character encoding range to which that character belongs.

[0187] In one implementation, after constructing the tree structure based on the character encoding range, the following steps are also included:

[0188] Store the tree structure in memory;

[0189] Real-time acquisition of text to be processed, including:

[0190] The text to be processed is acquired in real time and stored in memory.

[0191] Traverse the query tree structure to determine the character encoding range to which the character belongs, including:

[0192] In memory, the query tree structure is traversed to determine the character encoding range to which the character belongs.

[0193] In this application, after constructing a tree structure based on character encoding ranges, the constructed tree structure can be stored in memory for subsequent querying; moreover, the text to be processed can also be stored in memory to avoid redundant calculations and improve recognition efficiency.

[0194] In step S14, the characters are classified according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language.

[0195] In this step, the recognition results are classified. It can be understood that different languages ​​can be divided into two main categories: national languages ​​and minor languages. In this step, the characters are classified according to the target language corresponding to each character, and then these classification results are concatenated into a string for subsequent processing or output.

[0196] The categorized recognition results can be encapsulated into a specific format, such as JSON, XML, or a custom data structure, to facilitate subsequent data transmission, storage, or display. Furthermore, the encapsulated recognition results can be output, potentially meaning writing them to another data stream, database, file system, or other storage medium; the specific method is not limited.

[0197] like Figure 2 The diagram shown is a logical schematic of a method for identifying minority languages ​​based on an embodiment of this application. It includes:

[0198] Receive input data;

[0199] Data preprocessing is performed to remove duplicate characters and words. By deduplicating the text, redundancy in the data to be processed can be reduced, thus lowering computational complexity.

[0200] Perform data deduplication optimization to process input data into a more efficient data structure, ensuring the uniqueness of input data and access speed;

[0201] The optimized data storage stores the deduplicated and optimized input data in memory as input for text recognition, avoiding redundant calculations and improving recognition efficiency.

[0202] Then, we can perform minority language recognition, including input data validity checks, text recognition operator running status checks, data processing and whitelist checks, to obtain the text to be processed;

[0203] The text to be processed is subjected to minority language recognition, and a caching mechanism is used to optimize the process of recognizing consecutive characters.

[0204] Encapsulate the results of the recognition of minor languages ​​and output them.

[0205] As can be seen from the above, this application constructs a tree structure using the character encoding range of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing and querying the tree structure, and the recognition result can be obtained. This enables real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multilingual recognition needs in the real-time processing scenario of big data and improve the processing performance and application effect of text recognition.

[0206] The text recognition method provided in this application can be executed by a text recognition device. This application uses a text recognition device executing a terminal access method as an example to illustrate the device used in the text recognition method provided in this application.

[0207] Figure 3 This is a block diagram of a text recognition device according to an exemplary embodiment, comprising:

[0208] Construction module 201 is used to obtain the character encoding range of at least one preset language and construct a tree structure based on the character encoding range;

[0209] Module 202 is used to acquire the text to be processed in real time;

[0210] The traversal module 203 is used to traverse and query the tree structure for each character in the text to be processed, determine the character encoding interval to which the character belongs, and take the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character.

[0211] The output module 204 is used to classify the characters according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language.

[0212] Optionally, the construction module 201 is used for:

[0213] Establish a connection with a remote dictionary service;

[0214] Read the character encoding range of at least one preset language from the remote dictionary service.

[0215] Optionally, the construction module 201 is used for:

[0216] Create the first node of the tree structure, which serves as the root node;

[0217] Select a target interval from the character encoding interval of at least one of the preset languages, and use it as the node interval corresponding to the root node;

[0218] The character encoding intervals other than the target interval are traversed sequentially as candidate intervals. Based on the interval relationship between the candidate intervals and the node intervals corresponding to the tree structure, the position of the node corresponding to the candidate interval in the tree structure is determined, and a new node is created at the determined position. The candidate interval is used as the node interval corresponding to the new node to obtain the tree structure.

[0219] Optionally, the construction module 201 is further configured to:

[0220] The tree structure is stored in memory;

[0221] The acquisition module 202 is further configured to:

[0222] The text to be processed is acquired in real time and stored in the memory.

[0223] The traversal module 203 is specifically used for:

[0224] In the memory, the tree structure is traversed and queried to determine the character encoding range to which the character belongs.

[0225] Optionally, the acquisition module 202 is used to:

[0226] Obtain the input data and the configuration information of the input data;

[0227] Based on the configuration information, the text to be processed is extracted from the input data.

[0228] Optionally, the acquisition module 202 is used to:

[0229] Perform a data validity check on the input data;

[0230] If the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0231] Optionally, the acquisition module 202 is used to:

[0232] Obtain the running label of the text recognition operator;

[0233] When the running label indicates that the text recognition operator has started running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0234] Optionally, the text to be processed includes at least one dataset; the acquisition module 202 is used to:

[0235] Get the whitelist;

[0236] Detect whether the dataset is in the whitelist, and remove datasets not in the whitelist from the text to be processed.

[0237] Optionally, the acquisition module 202 is used to:

[0238] If the text to be processed is unstructured text, the text to be processed is deduplicated to obtain optimized text.

[0239] Optionally, the traversal module 203 is used for:

[0240] Each character in the text to be processed is taken as the target character in turn, and the character encoding range of the previous character of the target character is obtained as the cache range;

[0241] Determine whether the target character belongs to the cache range;

[0242] If the target character belongs to the cache range, then it is determined that the target character is the same as the character encoding range corresponding to the previous character of the target character;

[0243] If the target character does not belong to the cache range, the tree structure is traversed to determine the character encoding range to which the target character belongs.

[0244] Optionally, the traversal module 203 is used for:

[0245] Retrieve cached key-value pairs; the key of the cached key-value pair is the previous character of the target character, and the value of the cached key-value pair is the cache range;

[0246] After determining that the target character is the same as the character encoding range corresponding to the previous character of the target character, the method further includes:

[0247] Update the key of the cached key-value pair based on the target character;

[0248] After traversing the tree structure to determine the character encoding range to which the target character belongs, the method further includes:

[0249] Update the key of the cached key-value pair based on the target character;

[0250] Update the value of the cached key-value pair based on the character encoding range to which the target character belongs.

[0251] As can be seen from the above, this application constructs a tree structure using the character encoding range of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing and querying the tree structure, and the recognition result can be obtained. This enables real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multilingual recognition needs in the real-time processing scenario of big data and improve the processing performance and application effect of text recognition.

[0252] The text recognition method provided in this application can be executed by a terminal access terminal. This application uses the terminal access terminal executing the terminal access method as an example to illustrate the apparatus of the text recognition method provided in this application.

[0253] The text recognition device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0254] The text recognition device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0255] Optionally, such as Figure 4As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described text recognition method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0256] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0257] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0258] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0259] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0260] As can be seen from the above, this application constructs a tree structure using the character encoding range of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing and querying the tree structure, and the recognition result can be obtained. This enables real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multilingual recognition needs in the real-time processing scenario of big data and improve the processing performance and application effect of text recognition.

[0261] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0262] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0263] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0264] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described text recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0265] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0266] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described text recognition method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0267] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0268] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the text recognition method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0269] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0270] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0271] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A text recognition method, characterized in that, The method includes: Obtain the character encoding range of at least one preset language, and construct a tree structure based on the character encoding range; Real-time acquisition of text to be processed; For each character in the text to be processed, the tree structure is traversed and queried to determine the character encoding range to which the character belongs, and the preset language corresponding to the character encoding range to which the character belongs is taken as the target language corresponding to the character. Based on the target language corresponding to each character, the characters are classified to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language; The construction of the tree structure based on the character encoding range includes: Create the first node of the tree structure, which serves as the root node; Select a target interval from the character encoding interval of at least one of the preset languages, and use it as the node interval corresponding to the root node; The character encoding intervals other than the target interval are traversed sequentially as candidate intervals. Based on the interval relationship between the candidate intervals and the node intervals corresponding to the tree structure, the position of the node corresponding to the candidate interval in the tree structure is determined, and a new node is created at the determined position. The candidate interval is used as the node interval corresponding to the new node to obtain the tree structure. The step of traversing and querying the tree structure for each character in the text to be processed to determine the character encoding range to which the character belongs includes: Each character in the text to be processed is taken as the target character in turn, and the character encoding range of the previous character of the target character is obtained as the cache range; Determine whether the target character belongs to the cache range; If the target character belongs to the cache range, then it is determined that the target character is the same as the character encoding range corresponding to the previous character of the target character; If the target character does not belong to the cache range, the tree structure is traversed and queried to determine the character encoding range to which the target character belongs; The step of obtaining the character encoding range to which the previous character of the target character belongs, as a cache range, includes: Retrieve cached key-value pairs; the key of the cached key-value pair is the previous character of the target character, and the value of the cached key-value pair is the cache range; After determining that the target character is the same as the character encoding range corresponding to the previous character of the target character, the method further includes: Update the key of the cached key-value pair based on the target character; After traversing the tree structure to determine the character encoding range to which the target character belongs, the method further includes: Update the key of the cached key-value pair based on the target character; Update the value of the cached key-value pair based on the character encoding range to which the target character belongs.

2. The text recognition method according to claim 1, characterized in that, The step of obtaining the character encoding range of at least one preset language includes: Establish a connection with a remote dictionary service; Read the character encoding range of at least one preset language from the remote dictionary service.

3. The text recognition method according to claim 1, characterized in that, After constructing the tree structure based on the character encoding range, the method further includes: The tree structure is stored in memory; The real-time acquisition of the text to be processed includes: The text to be processed is acquired in real time and stored in the memory. The step of traversing and querying the tree structure to determine the character encoding range to which the character belongs includes: In the memory, the tree structure is traversed and queried to determine the character encoding range to which the character belongs.

4. The text recognition method according to claim 1, characterized in that, The real-time acquisition of the text to be processed includes: Obtain the input data and the configuration information of the input data; Based on the configuration information, the text to be processed is extracted from the input data.

5. The text recognition method according to claim 4, characterized in that, After obtaining the input data, the process includes: Perform a data validity check on the input data; If the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

6. The text recognition method according to claim 4, characterized in that, After obtaining the input data, the process includes: Obtain the running label of the text recognition operator; When the running label indicates that the text recognition operator has started running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

7. The text recognition method according to claim 1, characterized in that, The text to be processed includes at least one dataset; after acquiring the text to be processed in real time, the process includes: Get the whitelist; Detect whether the dataset is in the whitelist, and remove datasets not in the whitelist from the text to be processed.

8. The text recognition method according to claim 1, characterized in that, After acquiring the text to be processed in real time, the process also includes: If the text to be processed is unstructured text, the text to be processed is deduplicated to obtain optimized text.

9. A text recognition device, characterized in that, The device includes: A construction module is used to obtain the character encoding range of at least one preset language and construct a tree structure based on the character encoding range; The acquisition module is used to acquire the text to be processed in real time; The traversal module is used to traverse and query the tree structure for each character in the text to be processed, determine the character encoding interval to which the character belongs, and take the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character. The output module is used to classify the characters according to the target language corresponding to each character, and obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters in the text to be processed that correspond to the target language; The construction module is also used for: Create the first node of the tree structure, which serves as the root node; Select a target interval from the character encoding interval of at least one of the preset languages, and use it as the node interval corresponding to the root node; The character encoding intervals other than the target interval are traversed sequentially as candidate intervals. Based on the interval relationship between the candidate intervals and the node intervals corresponding to the tree structure, the position of the node corresponding to the candidate interval in the tree structure is determined, and a new node is created at the determined position. The candidate interval is used as the node interval corresponding to the new node to obtain the tree structure. The traversal module is used for: Each character in the text to be processed is taken as the target character in turn, and the character encoding range of the previous character of the target character is obtained as the cache range; Determine whether the target character belongs to the cache range; If the target character belongs to the cache range, then it is determined that the target character is the same as the character encoding range corresponding to the previous character of the target character; If the target character does not belong to the cache range, the tree structure is traversed and queried to determine the character encoding range to which the target character belongs; The traversal module is used for: Retrieve cached key-value pairs; the key of the cached key-value pair is the previous character of the target character, and the value of the cached key-value pair is the cache range; After determining that the target character is the same as the character encoding range corresponding to the previous character of the target character, the method further includes: Update the key of the cached key-value pair based on the target character; After traversing the tree structure to determine the character encoding range to which the target character belongs, the method further includes: Update the key of the cached key-value pair based on the target character; Update the value of the cached key-value pair based on the character encoding range to which the target character belongs.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the text recognition method as described in any one of claims 1-8.

11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the text recognition method as described in any one of claims 1-8.

Images