Text recognition method, device and equipment

Abstract

The invention discloses a text recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a character coding interval of at least one preset language, and constructing a tree structure based on the character coding interval; obtaining a to-be-processed text in real time; for each character in the to-be-processed text, traversing the query tree structure, determining a character coding interval to which the character belongs, and taking a preset language corresponding to the character coding interval to which the character belongs as a target language corresponding to the character; classifying the characters according to the target language corresponding to each character to obtain an identification result of the to-be-processed text; the recognition result is used for indicating the character corresponding to the target language included in the to-be-processed text. Compared with a traditional off-line batch processing mode, the real-time processing capability and the expansibility are enhanced, the multi-language recognition requirement in a big data real-time processing scene can be met, and the processing performance and the application effect of text recognition are improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and specifically relates to a text recognition method, device, equipment and storage medium. Background Art

[0002] With the increasing number of language exchanges between different countries and regions, the need for text recognition in minority languages ​​in real-time scenarios has become more urgent. For example, in applications such as social media monitoring, real-time translation, online customer service, and information security, fast and accurate recognition and processing of text in minority languages ​​is crucial for timely acquisition of key information and rapid response.

[0003] In the prior art, the minority language text recognition method mainly relies on the offline batch processing mode, which requires collecting a large amount of minority language text data in advance and then processing these minority language text data in batches.

[0004] However, this method can only be applied to the processing of static text data in small languages ​​that do not change or update frequently. When the data to be processed is real-time text data in small languages ​​that is constantly generated and needs to be processed immediately, the new text data in small languages ​​cannot be processed immediately and the results cannot be given. There is a processing delay. In other words, the traditional offline batch processing mode lacks real-time processing capabilities and cannot respond to small language recognition needs in a timely manner, resulting in poor timeliness and application effects of small language text recognition. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a text recognition method, apparatus, device and storage medium, which can solve the problem that the current small language text recognition method has insufficient real-time processing capability and cannot respond to small language recognition needs in a timely manner, resulting in poor timeliness and application effect of small language text recognition.

[0006] In a first aspect, an embodiment of the present application provides a text recognition method, the method comprising:

[0007] Obtaining a character encoding interval of at least one preset language, and constructing a tree structure based on the character encoding interval;

[0008] Get the text to be processed in real time;

[0009] For each character in the text to be processed, traverse and query the tree structure to determine the character encoding interval to which the character belongs, and use the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character;

[0010] According to the target language corresponding to each character, the characters are classified to obtain a recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

[0011] Optionally, obtaining the character encoding interval of at least one preset language includes:

[0012] Establish a connection with a remote dictionary service;

[0013] A character encoding interval of at least one preset language is read from the remote dictionary service.

[0014] Optionally, constructing a tree structure based on the character encoding interval includes:

[0015] Create the first node of the tree structure as the root node;

[0016] Selecting a target interval from the character encoding intervals of at least one of the preset languages ​​as a node interval corresponding to the root node;

[0017] The other character encoding intervals except the target interval are traversed in sequence as candidate intervals, and the positions of the nodes corresponding to the candidate intervals in the tree structure are determined according to the interval relationships between the candidate intervals and the node intervals corresponding to the tree structure, and a new node is created at the determined position, and the candidate interval is used as the node interval corresponding to the new node to obtain a tree structure.

[0018] Optionally, after constructing the tree structure based on the character encoding interval, the method further includes:

[0019] storing the tree structure in memory;

[0020] The real-time acquisition of the text to be processed includes:

[0021] Acquire the text to be processed in real time, and store the text to be processed in the memory;

[0022] The traversing and querying the tree structure to determine the character encoding interval to which the character belongs includes:

[0023] In the memory, the tree structure is traversed and queried to determine the character encoding interval to which the character belongs.

[0024] Optionally, the real-time acquisition of the text to be processed includes:

[0025] Obtaining input data and configuration information of the input data;

[0026] Based on the configuration information, text to be processed is extracted from the input data.

[0027] Optionally, after obtaining the input data, the step further includes:

[0028] Performing data validity check on the input data;

[0029] In the case where the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0030] Optionally, after obtaining the input data, the step further includes:

[0031] Get the running label of the text recognition operator;

[0032] In the case where the running tag indicates that the text recognition operator starts running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0033] Optionally, the text to be processed includes at least one data set; after the text to be processed is acquired in real time, the method includes:

[0034] Get the whitelist;

[0035] Detect whether the data set is in the white list, and delete the data set not included in the white list from the text to be processed.

[0036] Optionally, after obtaining the text to be processed in real time, the method further includes:

[0037] In the case that the text to be processed is an unstructured text, deduplication optimization is performed on the text to be processed to obtain an optimized text to be processed.

[0038] Optionally, for each character in the text to be processed, traversing and querying the tree structure to determine the character encoding interval to which the character belongs includes:

[0039] Taking each character in the text to be processed as a target character in turn, obtaining the character encoding interval to which the previous character of the target character belongs as a cache interval;

[0040] Determining whether the target character belongs to the cache interval;

[0041] If the target character belongs to the cache interval, determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same;

[0042] If the target character does not belong to the cache interval, the tree structure is traversed and queried to determine the character encoding interval to which the target character belongs.

[0043] Optionally, the obtaining, as a cache interval, a character encoding interval to which a previous character of the target character belongs includes:

[0044] Obtain a cache key-value pair; the key of the cache key-value pair is the previous character of the target character, and the value of the cache key-value pair is the cache interval;

[0045] After determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same, the method further includes:

[0046] Based on the target character, update the key of the cache key-value pair;

[0047] After traversing and querying the tree structure to determine the character encoding interval to which the target character belongs, the method further includes:

[0048] Based on the target character, update the key of the cache key-value pair;

[0049] Based on the character encoding interval to which the target character belongs, the value of the cache key-value pair is updated.

[0050] In a second aspect, an embodiment of the present application provides a text recognition device, the device comprising:

[0051] A construction module, used to obtain a character encoding interval of at least one preset language, and construct a tree structure based on the character encoding interval;

[0052] The acquisition module is used to acquire the text to be processed in real time;

[0053] A traversal module, for traversing and querying the tree structure for each character in the text to be processed, determining the character encoding interval to which the character belongs, and using the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character;

[0054] The output module is used to classify the characters according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

[0055] Optionally, the construction module is used to:

[0056] Establish a connection with a remote dictionary service;

[0057] A character encoding interval of at least one preset language is read from the remote dictionary service.

[0058] Optionally, the construction module is used to:

[0059] Create the first node of the tree structure as the root node;

[0060] Selecting a target interval from the character encoding intervals of at least one of the preset languages ​​as a node interval corresponding to the root node;

[0061] The other character encoding intervals except the target interval are traversed in sequence as candidate intervals, and the positions of the nodes corresponding to the candidate intervals in the tree structure are determined according to the interval relationships between the candidate intervals and the node intervals corresponding to the tree structure, and a new node is created at the determined position, and the candidate interval is used as the node interval corresponding to the new node to obtain a tree structure.

[0062] Optionally, the construction module is further used to:

[0063] storing the tree structure in memory;

[0064] The acquisition module is further used for:

[0065] Acquire the text to be processed in real time, and store the text to be processed in the memory;

[0066] The traversal module is specifically used for:

[0067] In the memory, the tree structure is traversed and queried to determine the character encoding interval to which the character belongs.

[0068] Optionally, the acquisition module is used to:

[0069] Obtaining input data and configuration information of the input data;

[0070] Based on the configuration information, text to be processed is extracted from the input data.

[0071] Optionally, the acquisition module is used to:

[0072] Performing data validity check on the input data;

[0073] In the case where the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0074] Optionally, the acquisition module is used to:

[0075] Get the running label of the text recognition operator;

[0076] In the case where the running tag indicates that the text recognition operator starts running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0077] Optionally, the text to be processed includes at least one data set; and the acquisition module is used to:

[0078] Get the whitelist;

[0079] Detect whether the data set is in the white list, and delete the data set not included in the white list from the text to be processed.

[0080] Optionally, the acquisition module is used to:

[0081] In the case that the text to be processed is an unstructured text, deduplication optimization is performed on the text to be processed to obtain an optimized text to be processed.

[0082] Optionally, the traversal module is used to:

[0083] Taking each character in the text to be processed as a target character in turn, obtaining the character encoding interval to which the previous character of the target character belongs as a cache interval;

[0084] Determining whether the target character belongs to the cache interval;

[0085] If the target character belongs to the cache interval, determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same;

[0086] If the target character does not belong to the cache interval, the tree structure is traversed and queried to determine the character encoding interval to which the target character belongs.

[0087] Optionally, the traversal module is used to:

[0088] Obtain a cache key-value pair; the key of the cache key-value pair is the previous character of the target character, and the value of the cache key-value pair is the cache interval;

[0089] After determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same, the method further includes:

[0090] Based on the target character, update the key of the cache key-value pair;

[0091] After traversing and querying the tree structure to determine the character encoding interval to which the target character belongs, the method further includes:

[0092] Based on the target character, update the key of the cache key-value pair;

[0093] Based on the character encoding interval to which the target character belongs, the value of the cache key-value pair is updated.

[0094] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0095] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0096] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0097] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0098] In the solution provided in the embodiment of the present application, a character coding interval of at least one preset language is obtained, and a tree structure is constructed based on the character coding interval; the text to be processed is obtained in real time; for each character in the text to be processed, the query tree structure is traversed to determine the character coding interval to which the character belongs, and the preset language corresponding to the character coding interval to which the character belongs is used as the target language corresponding to the character; the characters are classified according to the target language corresponding to each character to obtain a recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

[0099] As can be seen from the above, the present application proposes a text recognition method, which constructs a tree structure using character encoding intervals of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing the query tree structure to obtain the recognition result, thereby realizing real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multi-language recognition needs in the big data real-time processing scenario and improve the processing performance and application effect of text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 is a flowchart of a text recognition method according to an exemplary embodiment;

[0101] Figure 2 is a logic diagram of a minority language recognition method according to an exemplary embodiment;

[0102] Figure 3is a block diagram of a text recognition device according to an exemplary embodiment;

[0103] Figure 4 is a block diagram of an electronic device according to an exemplary embodiment;

[0104] Figure 5 The figure is a schematic diagram of a hardware structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0105] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0106] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0107] The text recognition method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0108] Figure 1 The flowchart of a text recognition method according to an exemplary embodiment is shown. The text recognition method includes the following steps.

[0109] In step S11, a character coding interval of at least one preset language is obtained, and a tree structure is constructed based on the character coding interval.

[0110] With the increasing language exchanges between different countries and regions, the demand for minority language text recognition in real-time scenarios has become more urgent. In related technologies, minority language text recognition methods mainly rely on offline batch processing mode, which requires pre-collecting a large amount of minority language text data and then processing these minority language text data in batches.

[0111] However, this method can only be applied to the processing of static text data in small languages ​​that do not change or update frequently. When the data to be processed is real-time text data in small languages ​​that is constantly generated and needs to be processed immediately, the new text data in small languages ​​cannot be processed immediately and the results cannot be given. There is a processing delay, and it is impossible to respond to small language recognition needs in a timely manner, resulting in poor timeliness and application effect of small language text recognition.

[0112] Based on this, this application proposes a text recognition method to solve the above problems, meet the multilingual recognition needs in big data real-time processing scenarios, and improve the processing performance and application effect of text recognition.

[0113] In this step, you first need to obtain the character encoding range (Unicode) of one or more preset languages, where the character encoding range is intended to set a unified and unique binary code for each character in each language to meet the needs of cross-language and cross-platform text conversion and processing. In other words, the characters included in each preset language will uniquely correspond to a part of the character encoding range, which may be continuous or divided into several segments, without specific limitation.

[0114] In one implementation, obtaining a character encoding range of at least one preset language includes:

[0115] Establish a connection with a remote dictionary service;

[0116] Read at least one character encoding range of a preset language from a remote dictionary service.

[0117] Among them, the Remote Dictionary Server (Redis) is used to store and manage character encoding intervals of various preset languages. It has efficient data access capabilities and supports dynamic updates and fast queries to ensure the accuracy and timeliness of text recognition.

[0118] That is to say, by setting and initializing the Redis connection, the character encoding range of each preset language can be read from Redis for subsequent character query and recognition.

[0119] After obtaining the character encoding interval of the preset language, a tree structure can be constructed based on the character encoding interval to facilitate subsequent interval query operations based on the tree structure. The tree structure is a nonlinear data structure consisting of a hierarchical set of n (n≥0) finite nodes, each of which contains a data element and several branches pointing to its subtrees.

[0120] In one implementation, constructing a tree structure based on character encoding intervals includes:

[0121] Create the first node of the tree structure as the root node;

[0122] Selecting a target interval from character encoding intervals of at least one preset language as a node interval corresponding to the root node;

[0123] Traverse the other character encoding intervals except the target interval in turn as candidate intervals, determine the position of the node corresponding to the candidate interval in the tree structure according to the interval relationship between the candidate interval and each node interval corresponding to the tree structure, create a new node at the determined position, and use the candidate interval as the node interval corresponding to the new node to obtain a tree structure.

[0124] In other words, the character encoding intervals read can be constructed into a tree structure so that efficient query can be achieved using the tree structure. First, a tree structure needs to be defined to store the character encoding intervals of each preset language. Each node in the tree structure represents a character encoding interval of a preset language. Then, each character encoding interval read from Redis needs to be inserted into the tree structure.

[0125] Among them, various types of tree structures can be selected to store character encoding intervals, such as binary search tree (BST), balanced binary search tree (such as AVL tree, red-black tree), B tree or B+ tree, etc. Each tree structure has its specific advantages and applicable scenarios.

[0126] For example, taking a binary search tree as an example, each node contains a character encoding range (defined by the starting character and the ending character) and pointers to the left subtree and the right subtree. Suppose the following character encoding ranges need to be inserted into the tree structure:

[0127] Range 1: [U+0041,U+005A] (uppercase letters A to Z)

[0128] Range 2: [U+0061,U+007A] (lowercase letters a to z)

[0129] Range 3: [U+03A0,U+03C1] (Greek letters Π to Ρ)

[0130] So, first create a new node with the starting character U+0041 and the ending character U+005A. Since the tree is empty, this node becomes the root node.

[0131] Then, create a new node with the starting character U+0061 and the ending character U+007A. Compare the starting character U+0061 of the new node with the starting character U+0041 of the root node. Since U+0061 is greater than U+0041, recursively insert the node at the right subtree position of the new node. At this point, the root node has no right subtree, so the new node becomes the right child of the root node.

[0132] After that, create a new node with the starting character U+03A0 and the ending character U+03C1. Compare the starting character U+03A0 of the new node with the starting character U+0041 of the root node. Since U+03A0 is greater than U+0041, recursively continue the comparison at the right subtree position of the new node. Compare U+03A0 with the starting character U+0061 of the right child node (interval 2).

[0133] Since U+03A0 is greater than U+0061, but interval 3 does not overlap with interval 2, the new node should become the right sibling node of interval 2. Insert the new node at the right subtree position of interval 2 (since interval 2 currently has no right subtree, the new node directly becomes its right child node).

[0134] The final tree structure is:

[0135] Root node: [U+0041,U+005A] (uppercase letters A to Z)

[0136] Right child node: [U+0061,U+007A] (lowercase letters a to z)

[0137] Right child: [U+03A0,U+03C1] (Greek letters Π to Ρ)

[0138] It can be understood that the tree structure is naturally suitable for representing hierarchical relationships, and can easily manage and query the hierarchical relationships of character encoding intervals; moreover, the tree structure enables the query operation of character encoding intervals to be completed within logarithmic time complexity, thereby improving query efficiency; in addition, through the tree structure, character intervals can be easily inserted, deleted and updated to meet the needs of dynamic changes in character encoding intervals.

[0139] In step S12, the text to be processed is obtained in real time.

[0140] In this step, the text to be processed can be obtained in real time. The text to be processed is the text that needs to be recognized. The data source of the text to be processed may include an online system, a real-time database, a data stream, etc., without specific limitation.

[0141] In this application, the specific execution of the text recognition operator can be implemented in the processElement method in ProcessFunction in Flink. Through Flink's stream processing framework, millisecond-level processing of the real-time acquired text to be processed can be achieved, which can quickly respond to real-time text recognition needs and ensure the timeliness and accuracy of data processing.

[0142] In one implementation, obtaining the text to be processed in real time includes:

[0143] Obtain input data and configuration information of the input data;

[0144] Based on the configuration information, the text to be processed is extracted from the input data.

[0145] That is to say, after obtaining the input data, it is necessary to further determine how to obtain the text to be processed based on the configuration information of the input data. For example, it may involve extracting specific fields from the input data, selecting specific data set codes, or selecting processing methods based on other conditions, etc., without specific limitation.

[0146] In one implementation, after obtaining the input data, the following steps are performed:

[0147] Perform data validity check on input data;

[0148] When the input data passes the data validity check, a step of extracting the text to be processed from the input data based on the configuration information is performed.

[0149] That is to say, after obtaining the input data, the input data needs to be checked for data validity. This is to ensure that the input data for subsequent processing is legal and conforms to the expected format. If the data is invalid, such as incorrect format, missing key fields, or containing illegal characters, it will be returned directly without subsequent processing to save system resources.

[0150] In one implementation, after obtaining the input data, the following steps are performed:

[0151] Get the running label of the text recognition operator;

[0152] When the run tag indicates that the text recognition operator starts running, the step of extracting the text to be processed from the input data based on the configuration information is executed.

[0153] That is to say, after obtaining the input data, it is necessary to further check whether the text recognition operator needs to be run. This step is a conditional judgment used to control whether the text recognition operator is executed. Among them, the situation in which the text recognition operator needs to be run usually depends on the specific application scenario and requirements.

[0154] Specifically, whether the text recognition operator is started can be controlled by a running tag. If the running tag is false, it means that the minority language recognition is not needed at present, then the input data is directly output and returned. If the running tag is true, it means that the minority language recognition is needed at present, then the subsequent steps can be continued.

[0155] In one implementation, the text to be processed includes at least one data set; after the text to be processed is acquired in real time, the method includes:

[0156] Get the whitelist;

[0157] Detect whether a data set is in the whitelist, and remove the data set not included in the whitelist from the text to be processed.

[0158] That is, after obtaining the text to be processed, you can obtain the whitelist, which is a predefined set that contains the data sets that need to be recognized. Then, you can check whether the data set included in the text to be processed is in the whitelist. If the current data set is in the whitelist, then continue to perform text recognition processing; if not, it may skip processing or perform other operations.

[0159] In one implementation, after obtaining the text to be processed in real time, the method further includes:

[0160] When the text to be processed is unstructured text, deduplication optimization is performed on the text to be processed to obtain an optimized text to be processed.

[0161] Specifically, the specific steps of deduplication optimization for the text to be processed may include but are not limited to:

[0162] Text preprocessing: Preprocess the text to be processed to remove repeated characters and words. By removing duplicates from the text, the redundancy of the text to be processed can be reduced and the computational complexity can be reduced.

[0163] Efficient data structure: Use efficient data structures (such as Set collections) to store the deduplicated text to be processed to ensure the uniqueness and access speed of the text to be processed.

[0164] In this application, the configuration parameters of the text recognition operator are set through parameter tools (such as ParameterTool) so that the behavior of the text recognition operator can be adjusted according to requirements at runtime. The configuration of the text recognition operator is mainly in the "open" method of the Flink rich function to ensure that the text recognition operator can run correctly in the subsequent processing.

[0165] For example, the configuration parameters of the text recognition operator may include but are not limited to:

[0166] Identification field list parameter: This parameter specifies which fields in the text to be processed need to be recognized. For example, if the text to be processed is a table containing multiple columns of information, this parameter will tell the operator which columns of text may need to be recognized.

[0167] Recognition result backfill field list parameter: After completing text recognition, it is usually necessary to save the recognition results back to a certain field. This parameter is used to specify which fields will be used to store the recognition results. For example, for each field that is recognized as a certain minority language, there can be a corresponding "result" field to store the specific recognized minority language information.

[0168] Recognition whitelist switch parameter: This is a switch parameter used to enable or disable the whitelist function of text recognition. If the whitelist is enabled, only the datasets or fields listed in the whitelist will be used for text recognition. This can be used to optimize performance or set only specific datasets to be processed.

[0169] Identification whitelist field list parameter: When the whitelist switch is enabled, this parameter specifies which fields (or data sets) are included in the whitelist. Only these fields of the text to be processed will be recognized for minority languages. This can be used to ensure that only specific data sets are processed without traversing the entire text to be processed.

[0170] In step S13, for each character in the text to be processed, the query tree structure is traversed to determine the character encoding interval to which the character belongs, and the preset language corresponding to the character encoding interval to which the character belongs is used as the target language corresponding to the character.

[0171] In this step, for each character in the text to be processed, it is necessary to traverse the query tree structure, determine the character coding interval to which the character belongs based on the interval relationship between each node in the tree structure and the character coding interval corresponding to each node, and then use the preset language corresponding to the character coding interval to which the character belongs as the target language corresponding to the character.

[0172] In one implementation, for each character in the text to be processed, a query tree structure is traversed to determine the character encoding interval to which the character belongs, including:

[0173] Taking each character in the text to be processed as the target character in turn, obtaining the character encoding interval to which the previous character of the target character belongs as the cache interval;

[0174] Determine whether the target character belongs to the cache interval;

[0175] If the target character belongs to the cache interval, it is determined that the character encoding interval corresponding to the target character and the character preceding the target character is the same;

[0176] If the target character does not belong to the cache interval, the query tree structure is traversed to determine the character encoding interval to which the target character belongs.

[0177] That is to say, when processing consecutive characters, it is assumed that the next character belongs to the same target language as the previous character. Then, first check whether there is a cache interval of the previous character in the cache. If so, first search in the cache whether the target character belongs to the same cache interval as the previous character.

[0178] If the cache hits, that is, the target character belongs to the cache interval, the result is returned directly, avoiding the need to traverse and recognize the tree structure again, which significantly improves the recognition speed. If the cache misses, that is, the target character does not belong to the cache interval, the recognition is performed again from the tree structure.

[0179] In one implementation, obtaining the character encoding interval to which the previous character of the target character belongs as the cache interval includes:

[0180] Get the cache key-value pair; the key of the cache key-value pair is the previous character of the target character, and the value of the cache key-value pair is the cache interval;

[0181] After determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same, the following further includes:

[0182] Based on the target character, update the key of the cache key-value pair;

[0183] After traversing the query tree structure and determining the character encoding interval to which the target character belongs, the following steps are also included:

[0184] Based on the target character, update the key of the cache key-value pair;

[0185] Update the value of the cache key-value pair based on the character encoding range to which the target character belongs.

[0186] That is to say, in order to improve the efficiency of continuous character recognition, the Caffeine cache mechanism can be used. Caffeine is a high-performance Java cache library that can effectively improve the cache hit rate and reduce repeated calculations. Among them, the Caffeine cache library can be used to design a cache structure for storing the previous character and its corresponding character encoding interval. Among them, the cache structure can be expressed in the form of a cache key-value pair (Key-Value), the key (Key) is the last recognized character, and the value (Value) is the character encoding interval to which the character belongs.

[0187] In one implementation, after constructing the tree structure based on the character encoding interval, the method further includes:

[0188] Store the tree structure in memory;

[0189] Get the text to be processed in real time, including:

[0190] Obtain the text to be processed in real time and store it in memory;

[0191] Traverse the query tree structure to determine the character encoding range to which the character belongs, including:

[0192] In memory, traverse the query tree structure to determine the character encoding range to which the character belongs.

[0193] In the present application, after constructing a tree structure based on the character encoding interval, the constructed tree structure can be stored in the memory for subsequent query use; moreover, the text to be processed can also be stored in the memory to avoid repeated calculations and improve recognition efficiency.

[0194] In step S14, the characters are classified according to the target language corresponding to each character to obtain a recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

[0195] In this step, the classification of recognition results is realized. It can be understood that different languages ​​can be divided into two categories: national languages ​​and minority languages. In this step, the characters are classified according to the target language corresponding to each character, and then these classification results are concatenated into a character string for subsequent processing or output.

[0196] The classified recognition results can be packaged into a specific format, such as JSON, XML or a custom data structure, to facilitate subsequent data transmission, storage or display. Furthermore, the packaged recognition results can be output, which may mean writing the recognition results to another data stream, database, file system or other storage medium, without specific limitation.

[0197] like Figure 2 FIG. 1 is a logic diagram of a minority language recognition method based on an embodiment of the present application. The method includes:

[0198] Receive input data;

[0199] Perform data preprocessing to remove duplicate characters and words. By removing duplicate text, the redundancy of the data to be processed can be reduced and the computational complexity can be lowered;

[0200] Perform data deduplication optimization and process input data into a more efficient data structure to ensure the uniqueness and access speed of input data;

[0201] Optimized data storage: The deduplicated and optimized input data is stored in memory as input for text recognition, avoiding repeated calculations and improving recognition efficiency.

[0202] Then, minority language recognition can be performed, including input data validity check, text recognition operator operation status check, data processing and whitelist check to obtain the text to be processed;

[0203] The text to be processed is recognized in minority languages. In the process of minority language recognition, a cache mechanism is used for optimization in order to improve the efficiency of continuous character recognition.

[0204] Encapsulate the minority language recognition results and output them.

[0205] As can be seen from the above, the present application constructs a tree structure using character encoding intervals of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing the query tree structure to obtain the recognition result, thereby realizing real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multi-language recognition needs in the big data real-time processing scenario and improve the processing performance and application effect of text recognition.

[0206] The text recognition method provided in the embodiment of the present application can be executed by a text recognition device. In the embodiment of the present application, a method in which a text recognition device executes terminal access is taken as an example to illustrate the device of the text recognition method provided in the embodiment of the present application.

[0207] Figure 3 is a block diagram of a text recognition device according to an exemplary embodiment, comprising:

[0208] A construction module 201 is used to obtain a character encoding interval of at least one preset language and construct a tree structure based on the character encoding interval;

[0209] An acquisition module 202 is used to acquire the text to be processed in real time;

[0210] A traversal module 203 is used to traverse and query the tree structure for each character in the text to be processed, determine the character encoding interval to which the character belongs, and use the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character;

[0211] The output module 204 is used to classify the characters according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

[0212] Optionally, the construction module 201 is used to:

[0213] Establish a connection with a remote dictionary service;

[0214] A character encoding interval of at least one preset language is read from the remote dictionary service.

[0215] Optionally, the construction module 201 is used to:

[0216] Create the first node of the tree structure as the root node;

[0217] Selecting a target interval from the character encoding intervals of at least one of the preset languages ​​as a node interval corresponding to the root node;

[0218] The other character encoding intervals except the target interval are traversed in sequence as candidate intervals, and the positions of the nodes corresponding to the candidate intervals in the tree structure are determined according to the interval relationships between the candidate intervals and the node intervals corresponding to the tree structure, and a new node is created at the determined position, and the candidate interval is used as the node interval corresponding to the new node to obtain a tree structure.

[0219] Optionally, the construction module 201 is further used to:

[0220] storing the tree structure in memory;

[0221] The acquisition module 202 is further used for:

[0222] Acquire the text to be processed in real time, and store the text to be processed in the memory;

[0223] The traversal module 203 is specifically used for:

[0224] In the memory, the tree structure is traversed and queried to determine the character encoding interval to which the character belongs.

[0225] Optionally, the acquisition module 202 is used to:

[0226] Obtaining input data and configuration information of the input data;

[0227] Based on the configuration information, text to be processed is extracted from the input data.

[0228] Optionally, the acquisition module 202 is used to:

[0229] Performing data validity check on the input data;

[0230] In the case where the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0231] Optionally, the acquisition module 202 is used to:

[0232] Get the running label of the text recognition operator;

[0233] In the case where the running tag indicates that the text recognition operator starts running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

[0234] Optionally, the text to be processed includes at least one data set; the acquisition module 202 is used to:

[0235] Get the whitelist;

[0236] Detect whether the data set is in the white list, and delete the data set not included in the white list from the text to be processed.

[0237] Optionally, the acquisition module 202 is used to:

[0238] In the case that the text to be processed is an unstructured text, deduplication optimization is performed on the text to be processed to obtain an optimized text to be processed.

[0239] Optionally, the traversal module 203 is used to:

[0240] Taking each character in the text to be processed as a target character in turn, obtaining the character encoding interval to which the previous character of the target character belongs as a cache interval;

[0241] Determining whether the target character belongs to the cache interval;

[0242] If the target character belongs to the cache interval, determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same;

[0243] If the target character does not belong to the cache interval, the tree structure is traversed and queried to determine the character encoding interval to which the target character belongs.

[0244] Optionally, the traversal module 203 is used to:

[0245] Obtain a cache key-value pair; the key of the cache key-value pair is the previous character of the target character, and the value of the cache key-value pair is the cache interval;

[0246] After determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same, the method further includes:

[0247] Based on the target character, update the key of the cache key-value pair;

[0248] After traversing and querying the tree structure to determine the character encoding interval to which the target character belongs, the method further includes:

[0249] Based on the target character, update the key of the cache key-value pair;

[0250] Based on the character encoding interval to which the target character belongs, the value of the cache key-value pair is updated.

[0251] As can be seen from the above, the present application constructs a tree structure using character encoding intervals of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing the query tree structure to obtain the recognition result, thereby realizing real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multi-language recognition needs in the big data real-time processing scenario and improve the processing performance and application effect of text recognition.

[0252] The text recognition method provided in the embodiment of the present application can be executed by a terminal access terminal. In the embodiment of the present application, a method in which a terminal access terminal executes terminal access is taken as an example to illustrate the apparatus of the text recognition method provided in the embodiment of the present application.

[0253] The text recognition device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0254] The text recognition device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0255] Alternatively, if Figure 4As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned text recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0256] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0257] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0258] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0259] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0260] As can be seen from the above, the present application constructs a tree structure using character encoding intervals of at least one preset language. After obtaining the real-time text to be processed, the target language included in the text to be processed can be identified by traversing the query tree structure to obtain the recognition result, thereby realizing real-time recognition of the text to be processed. Compared with the traditional offline batch processing mode, the real-time processing capability and scalability are enhanced, which can meet the multi-language recognition needs in the big data real-time processing scenario and improve the processing performance and application effect of text recognition.

[0261] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0262] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0263] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0264] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned text recognition method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0265] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0266] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned text recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0267] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0268] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned text recognition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0269] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0270] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0271] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A text recognition method, characterized in that: The method comprises: Obtaining a character encoding interval of at least one preset language, and constructing a tree structure based on the character encoding interval; Get the text to be processed in real time; For each character in the text to be processed, traverse and query the tree structure to determine the character encoding interval to which the character belongs, and use the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character; According to the target language corresponding to each character, the characters are classified to obtain a recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

2. The text recognition method according to claim 1, characterized in that: The step of obtaining a character encoding range of at least one preset language includes: Establish a connection with a remote dictionary service; A character encoding interval of at least one preset language is read from the remote dictionary service.

3. The text recognition method according to claim 1, characterized in that: The constructing a tree structure based on the character encoding interval includes: Create the first node of the tree structure as the root node; Selecting a target interval from the character encoding intervals of at least one of the preset languages ​​as a node interval corresponding to the root node; The other character encoding intervals except the target interval are traversed in sequence as candidate intervals, and the positions of the nodes corresponding to the candidate intervals in the tree structure are determined according to the interval relationships between the candidate intervals and the node intervals corresponding to the tree structure, and a new node is created at the determined position, and the candidate interval is used as the node interval corresponding to the new node to obtain a tree structure.

4. The text recognition method according to claim 1, characterized in that: After constructing the tree structure based on the character encoding interval, the method further includes: storing the tree structure in memory; The real-time acquisition of the to-be-processed text includes: Acquire the text to be processed in real time, and store the text to be processed in the memory; The traversing and querying the tree structure to determine the character encoding interval to which the character belongs includes: In the memory, the tree structure is traversed and queried to determine the character encoding interval to which the character belongs.

5. The text recognition method according to claim 1, characterized in that: The real-time acquisition of the to-be-processed text includes: Obtaining input data and configuration information of the input data; Based on the configuration information, text to be processed is extracted from the input data.

6. The text recognition method according to claim 5, characterized in that: After the input data is obtained, the following steps are included: Performing data validity check on the input data; In the case where the input data passes the data validity check, the step of extracting the text to be processed from the input data based on the configuration information is performed.

7. The text recognition method according to claim 5, characterized in that: After the input data is obtained, the following steps are included: Get the running label of the text recognition operator; In the case where the running tag indicates that the text recognition operator starts running, the step of extracting the text to be processed from the input data based on the configuration information is performed.

8. The text recognition method according to claim 1, characterized in that: The text to be processed includes at least one data set; after the text to be processed is acquired in real time, the method includes: Get the whitelist; Detect whether the data set is in the white list, and delete the data set not included in the white list from the text to be processed.

9. The text recognition method according to claim 1, characterized in that: After the real-time acquisition of the to-be-processed text, the method further includes: In the case that the text to be processed is an unstructured text, deduplication optimization is performed on the text to be processed to obtain an optimized text to be processed.

10. The text recognition method according to claim 1, characterized in that: The step of traversing and querying the tree structure for each character in the to-be-processed text to determine the character encoding interval to which the character belongs includes: Taking each character in the text to be processed as a target character in turn, obtaining the character encoding interval to which the previous character of the target character belongs as a cache interval; Determining whether the target character belongs to the cache interval; If the target character belongs to the cache interval, determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same; If the target character does not belong to the cache interval, the tree structure is traversed and queried to determine the character encoding interval to which the target character belongs.

11. The text recognition method according to claim 10, characterized in that: The step of obtaining the character encoding interval to which the previous character of the target character belongs as a cache interval includes: Obtain a cache key-value pair; the key of the cache key-value pair is the previous character of the target character, and the value of the cache key-value pair is the cache interval; After determining that the target character and the character encoding interval corresponding to the previous character of the target character are the same, the method further includes: Based on the target character, update the key of the cache key-value pair; After traversing and querying the tree structure to determine the character encoding interval to which the target character belongs, the method further includes: Based on the target character, update the key of the cache key-value pair; Based on the character encoding interval to which the target character belongs, the value of the cache key-value pair is updated.

12. A text recognition device, characterized in that: The device comprises: A construction module, used to obtain a character encoding interval of at least one preset language, and construct a tree structure based on the character encoding interval; The acquisition module is used to acquire the text to be processed in real time; A traversal module, for traversing and querying the tree structure for each character in the text to be processed, determining the character encoding interval to which the character belongs, and using the preset language corresponding to the character encoding interval to which the character belongs as the target language corresponding to the character; The output module is used to classify the characters according to the target language corresponding to each character to obtain the recognition result of the text to be processed; the recognition result is used to indicate the characters corresponding to the target language included in the text to be processed.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the text recognition method as described in any one of claims 1 to 11.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the text recognition method according to any one of claims 1 to 11 are implemented.

Images