Text exception processing method and device, equipment, storage medium and product

Abstract

The application discloses a text exception processing method and device, equipment, storage medium and product, and relates to the technical field of text processing and image recognition. The text exception processing method comprises: performing first-level identification on a text to be processed to obtain a first identification result comprising semantic text, and inputting the semantic text after sampling into a large language model to identify text exceptions by combining the semantics of the text with the large language model. The application performs partial sampling before inputting the text into the large language model, avoids processing all the text by the large language model, and reduces the cost of text exception identification. Moreover, the application samples the semantic text obtained by identification, so that the complete semantics of the text can be combined with the large language model to identify exceptions, the misjudgment and missed judgment of text garbled codes are avoided, and the accuracy of text exception identification is improved. Therefore, the application can balance the accuracy and cost of text exception identification.

Description

Technical Field

[0001] This application relates to the fields of text processing and image recognition technology, and in particular to text anomaly processing methods, apparatus, devices, storage media and products. Background Technology

[0002] In document digitization, knowledge base construction, and multilingual document translation pipelines, high-quality and accurate extraction of text content from original documents is a core prerequisite. Due to the complex and diverse sources of documents in real-world scenarios, the text extraction process is highly susceptible to various physical and logical anomalies, necessitating effective identification of these anomalies.

[0003] Currently, regular expressions or frequency recognition methods are mainly used to determine whether there are garbled characters or other anomalies in the extracted documents. However, using only regular expressions or frequency recognition to identify garbled characters in extracted text is prone to misidentifying meaningful characters as garbled characters, or misidentifying garbled text with a few normal characters as normal text, resulting in low accuracy of text anomaly identification. On the other hand, calling a large language model for each page of text will incur significant overhead. Therefore, current text extraction processing methods cannot balance the accuracy and cost of text anomaly identification. Summary of the Invention

[0004] The main objective of this application is to provide a text anomaly processing method, apparatus, device, storage medium, and product, aiming to solve the technical problem of being unable to balance the accuracy and cost of text anomaly recognition.

[0005] To achieve the above objectives, this application proposes a text anomaly handling method, the method comprising: Get the text to be processed; The text to be processed is input into a preset pre-text processing model. Based on the pre-text processing model, the text to be processed is subjected to a first-level recognition, and a first recognition result is output. The first recognition result includes at least the corresponding semantic text. A portion of the text is sampled from the semantic text and input into a preset large language model. The large language model performs a second-level recognition on the portion of the text to output the corresponding target anomaly result.

[0006] Optionally, the step of sampling a portion of the text from the semantic text includes: Dynamically sample portions of text from the semantic text.

[0007] Optionally, the step of dynamically sampling a portion of the text from the semantic text includes: Generate random numbers for different paragraphs of the semantic text; Based on the comparison result between the random number and the preset dynamic sampling factor, it is determined whether to include the corresponding paragraph as part of the text to be sampled.

[0008] Optionally, the step of determining whether to include the corresponding paragraph as part of the sampled text based on the comparison result between the random number and the preset dynamic sampling factor includes: If the random number is less than the dynamic sampling factor, then the corresponding paragraph will be used as the sampled part of the text.

[0009] Optionally, the step of performing a second-level recognition on the portion of text using the large language model to output the corresponding target anomaly result includes: The large language model is used to perform a second-level recognition on the partial text, and a second recognition result is output. Based on the second identification result, the first identification result is corrected in reverse, and the corrected first identification result is taken as the target abnormal result.

[0010] Optionally, the step of reversing the first identification result based on the second identification result and using the corrected first identification result as the target abnormal result includes: Based on the second recognition result, if it is determined that the part of the text is not represented using a preset language, then the first recognition result is corrected to be non-garbled text.

[0011] Optionally, the step of performing a second-level recognition on the portion of text using the large language model to output the corresponding target anomaly result further includes: The large language model is used to perform a second-level recognition on the text. If the second-level recognition times out or is abnormal, the first recognition result is taken as the target abnormal result.

[0012] Optionally, the step of performing second-level recognition on the portion of text using the large language model includes: Based on the large language model and preset constraint rules, a second-level recognition is performed on the partial text. The preset constraint rules include at least one of the following: mapping garbled character processing rules, special character processing rules, local garbled character processing rules, and identifier processing rules.

[0013] Optionally, after the step of performing a second-level recognition on the portion of text using the large language model to output the corresponding target anomaly result, the method further includes: If the result of the text anomaly processing is garbled text, then the text image corresponding to the text to be processed is determined; The text image is reconstructed by secondary mapping to obtain the reconstructed text, and the reconstructed text is then processed.

[0014] Optionally, the step of performing secondary mapping reconstruction on the text image to obtain the reconstructed text includes: Determine the scaling factor and rotation angle of the page where the text image is located; The position of the text corresponding to the text image on the page is determined based on the scaling factor, rotation angle, and image edges of the text image.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a search suggestion device, the search suggestion device comprising: The text acquisition module is used to acquire the text to be processed. The first recognition module is used to input the text to be processed into a preset pre-text processing model, perform first-level recognition on the text to be processed based on the pre-text processing model, and output a first recognition result, wherein the first recognition result includes at least the corresponding semantic text; The second recognition module is used to sample a portion of the text from the semantic text and input the portion of the text into a preset large language model. The large language model is used to perform a second-level recognition on the portion of the text to output the corresponding target anomaly result.

[0016] Optionally, the second recognition module is further configured to dynamically sample portions of text from the semantic text.

[0017] Optionally, the second identification module is further configured to generate random numbers for different paragraphs of the semantic text; The second identification module is further configured to determine whether to include the corresponding paragraph as part of the text to be sampled based on the comparison result between the random number and the preset dynamic sampling factor.

[0018] Optionally, the second identification module is further configured to, if the random number is less than the dynamic sampling factor, use the corresponding paragraph as the sampled part of the text.

[0019] Optionally, the second recognition module is further configured to perform a second-level recognition on the portion of text using the large language model and output a second recognition result; The second identification module is further configured to correct the first identification result in reverse based on the second identification result, and use the corrected first identification result as the target abnormal result.

[0020] Optionally, the second recognition module is further configured to, based on the second recognition result, correct the first recognition result to be non-garbled text if it is determined that the part of the text is not represented using a preset language.

[0021] Optionally, the second recognition module is further configured to perform a second-level recognition on the partial text using the large language model. If the second-level recognition times out or is abnormal, the first recognition result is taken as the target abnormal result.

[0022] In addition, to achieve the above objectives, this application also proposes a text anomaly processing device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the text anomaly processing method as described above.

[0023] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the text exception handling method described above.

[0024] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the text exception handling method described above.

[0025] One or more technical solutions proposed in this application have at least the following technical effects: This application employs a two-level recognition process for the text to be processed. In the first level, it identifies a first recognition result including semantic text. This semantic text is then sampled and input into a large language model. The large language model, combined with the semantics of the text, identifies text anomalies. Because this application performs partial sampling before inputting the text into the large language model, it avoids processing the entire text through the large language model, reducing the overhead of text anomaly recognition. Furthermore, since the partial sampling is performed on the identified semantic text, anomaly recognition can be performed by combining the complete semantics of the text with the large language model. This avoids misidentifying meaningful characters as gibberish or misidentifying gibberish text with a few normal characters as normal, thereby improving the accuracy of text anomaly recognition. Therefore, this application can balance the accuracy and cost of text anomaly recognition. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating an embodiment of the text anomaly handling method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the text anomaly handling method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the text anomaly handling method of this application; Figure 4 This is a schematic diagram of the overall text anomaly handling process provided in Embodiment 3 of the text anomaly handling method of this application; Figure 5 This is a schematic diagram of the module structure of the text exception handling device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the text exception handling method in the embodiments of this application.

[0029] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0033] Additionally, it should be noted that in this article, "return" means returning to a specific step and re-executing from that step.

[0034] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, may be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or," "and / or," "including at least one of the following," etc., as used in this application, may be interpreted as inclusive, or mean any one or any combination thereof. For example, "including at least one of the following: A, B, C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Similarly, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0035] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0036] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0037] It should be noted that step designations such as S10 and S20 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S20 first and then S10, etc., but these should all be within the protection scope of this application.

[0038] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0039] Based on this, embodiments of this application provide a text anomaly handling method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the text anomaly handling method of this application.

[0040] Understandably, extracting high-quality and accurate text content from original documents is a core prerequisite in document digitization, knowledge base construction, and multilingual document translation pipelines. Due to the complex and diverse sources of documents in real-world scenarios, the text extraction process is highly susceptible to various physical and logical anomalies. Specific types of anomalies may include: 1. Character encoding and font mapping anomalies (Garbled Text): A large number of PDF (Portable Document Format) documents have missing font embedding, use non-standard / custom ToUnicode CMap (character mapping table) for embedded fonts, lack of dedicated Symbol (symbol font) / Wingdings (symbol font family) fonts in the system, or have anti-copy protection set up, resulting in the extracted text appearing as discontinuous garbled characters, illegal characters or meaningless symbols.

[0041] 2. Text layer and image layer overlap and watermark noise: Some scanned or damaged documents have blurry vector watermarks under the text layer, or the text extraction layer contains a large number of software-generated, semantically repetitive noise markers.

[0042] Traditional methods for document extraction anomaly detection involve matching high-frequency word lists, the proportion of non-printed characters, or using regular expressions to match the extracted text. However, these methods often misclassify normal text as garbled when processing documents containing specific technical symbols, engineering drawing codes (such as "WS-2025-X-001"), invoices, long strings of certificate numbers, and mathematical formulas. Conversely, they tend to correctly identify genuine font-mapped garbled text mixed with a small number of Chinese characters / common words. Furthermore, calling a large language model for every page of text incurs significant costs. Therefore, current text extraction methods cannot simultaneously achieve both high accuracy and low cost in anomaly detection.

[0043] To overcome the aforementioned shortcomings, this embodiment performs two-level recognition on the text to be processed. In the first-level recognition, a first recognition result including semantic text is obtained, ensuring that the recognized text possesses complete semantics. This embodiment then samples the text with complete semantics and inputs it into a large language model, allowing the large language model to process only a portion of the text with complete semantics. Therefore, this embodiment reduces the cost of processing all text by partially sampling the model, and since the input text contains complete semantics, the large language model can analyze whether the text is garbled based on semantics, improving the accuracy of text anomaly detection. This approach balances the cost and accuracy of text anomaly detection.

[0044] In this embodiment, the text anomaly handling method includes steps S10 to S30: Step S10: Obtain the text to be processed; It should be noted that the execution subject of this embodiment may be a data stream processing device with data processing, network communication and program running functions, such as a server, or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0045] It is understood that the text to be processed in this embodiment may come from various business scenarios, such as text content extracted from PDF, Word and other documents, text recognized by OCR from images, or plain text data directly pushed by upstream systems.

[0046] In this embodiment, the method of acquiring the text to be processed can be flexibly configured to adapt to different application scenarios and data scales. The specific data acquisition method can be to continuously acquire the text data generated by the upstream system in a streaming manner by accessing a distributed message queue or real-time data stream platform, or to read the text files to be processed in batches through a file system interface or object storage service. No limitation is made here.

[0047] In this embodiment, a standardized error code system is used during the data reading process. If file encryption or reading failure occurs, an error is reported and an error code such as "1012 File Encryption" or "4004 Read Failure" is output, so as to trace the abnormal file reading.

[0048] Step S20: Input the text to be processed into a preset pre-text processing model, perform first-level recognition on the text to be processed based on the pre-text processing model, and output a first recognition result, wherein the first recognition result includes at least the corresponding semantic text; Understandably, since the text to be processed can be text with complete semantics, or text without complete semantics such as mathematical formulas, special characters, engineering codes, and tables, if text without complete semantics is input into the large language model, the large language model will not be able to accurately determine whether the text is garbled.

[0049] Therefore, this embodiment performs preliminary identification on the text to be processed to obtain a first identification result containing at least the corresponding semantic text, so that the text input into the large language model is all text with complete semantics. Thus, text anomaly identification can be performed by combining the semantics of the text with the large language model, thereby improving the accuracy of text anomaly identification.

[0050] In a specific implementation, the first recognition result of this embodiment may contain only semantic text, or it may contain semantic text and a preliminary text anomaly judgment result. Based on the aforementioned pre-processing text model, this embodiment performs the first-level recognition of the text to be processed in the following ways: 1. Determine garbled characters and characters lacking complete semantic meaning in the text using preset matching rules, delete the garbled characters and characters lacking complete semantic meaning, and obtain a first recognition result containing the corresponding semantic text; 2. Based on the coordinate positions of the characters in the text to be processed, divide the text into multiple different text segments, determine whether each text segment has complete semantic meaning, and combine all text segments with complete semantic meaning to obtain the first recognition result.

[0051] Furthermore, the first recognition result of this embodiment may also include the language determination result of the text. In a specific implementation, if Chinese and English are used as preset languages, the steps for language determination in this embodiment are as follows: 1. Extract the first N (e.g., 1000) characters of the text; 2. Use regular expressions in a script to count the number of Chinese characters (chineseCharCount); 3. Use lexical regular expressions to extract consecutive valid English words and calculate the total number of characters in each English word: englishWordCharCount; 4. Calculate the percentage of Chinese characters: chineseCharRatio = chineseCharCount / allCharCount and the percentage of English words: englishWordCharRatio = englishWordCharCount / allCharCount; 5. When the proportion of English words is >0.5, the predicted language is English; for Chinese, when there are no English words, the threshold for the proportion of Chinese words is set to >0.2, and when there are English words, the threshold is set to >0.5. When these conditions are met, the predicted language is Chinese.

[0052] Specifically, the thresholds for preset language and language type judgment in the above judgment process can be set according to requirements.

[0053] Step S30: Sample a portion of the text from the semantic text and input the portion of the text into a preset large language model. Perform a second-level recognition on the portion of the text through the large language model to output the corresponding target anomaly result.

[0054] Understandably, in scenarios involving the processing of millions of pages of high-concurrency documents, even processing only text with complete semantics can still incur significant costs. Therefore, this embodiment samples the semantic text before inputting it into the large language model. By sampling, the large language model only needs to process a portion of the text, thereby reducing costs.

[0055] Furthermore, in this embodiment, the sampled text is input into the large language model. Since the input text is obtained by sampling semantic text, the large language model in this embodiment can combine the semantics of the text to judge text anomalies, avoiding misjudgments and omissions caused by fixed rule matching, thereby improving the accuracy of text anomaly recognition.

[0056] In a specific implementation, the method for sampling text in this embodiment can be as follows: 1. Based on the coordinates of the semantic text, a certain number of sampling coordinates are randomly generated, and the semantic text at the sampling coordinates is used as the sampled part of the text; 2. A feature vector is generated for each semantic text, and the text with a larger difference between the feature vectors is used as the sampled text, until the preset number of samples is met.

[0057] Furthermore, in order to achieve sampling of semantic text, a specific implementation method for sampling a portion of the text from the semantic text may also be: Dynamically sample portions of text from the semantic text.

[0058] Understandably, in practical applications, the total amount of text data to be processed will constantly change. If sampling is performed in a fixed manner, it is easy to end up with too much or too little text sampled. Therefore, this embodiment adjusts the number of samples through dynamic sampling to obtain a suitable portion of the sampled text.

[0059] In a specific implementation, this embodiment can automatically adjust the sampling ratio according to the amount of text to be processed, and sample by different sampling ratios. Alternatively, different text length factors can be set to sample text of corresponding lengths based on the text length factors.

[0060] Furthermore, in order to achieve dynamic sampling of semantic text, a specific implementation method for dynamically sampling a portion of the text from the semantic text may also be: Generate random numbers for different paragraphs of the semantic text; Based on the comparison result between the random number and the preset dynamic sampling factor, it is determined whether to include the corresponding paragraph as part of the text to be sampled.

[0061] Understandably, when controlling the input size of a large language model, sampling text in a fixed manner can easily lead to abnormal distribution of the sampled text. For example, the sampled text might be concentrated at the beginning or end of the text, causing anomalies in other regions of the text to be processed to be missed. Therefore, this embodiment generates a random number for each semantic text segment and compares the random number with an adjustable dynamic sampling factor to determine whether to include the corresponding segment as part of the sampled text. Thus, while controlling the input size of the large language model through dynamic sampling, the generation of random numbers ensures that the sampled text includes segments from various positions within the text, thereby enabling the identification of text anomalies at different locations.

[0062] In specific implementation, this embodiment can use the corresponding text segment as part of the sampled text when the random number is higher than the sampling factor, or use the corresponding text segment as part of the sampled text when the random number is lower than the sampling factor. It can also calculate the ratio of the sampling factor to the random number, and use the corresponding text segment as part of the sampled text when the ratio is lower than a preset threshold.

[0063] Furthermore, the specific implementation of the step of determining whether to include the corresponding paragraph as part of the sampled text based on the comparison result of the random number and the preset dynamic sampling factor in this embodiment can also be: If the random number is less than the dynamic sampling factor, then the corresponding paragraph will be used as the sampled part of the text.

[0064] It should be noted that in this embodiment, the dynamic sampling factor P is obtained from the system cloud control configuration RT_LLM_GARBLED_CHECK (real-time large language model garbled character detection). The function for generating random numbers and comparison is enableDetection = Math.random()<= P / 100 (randomly deciding whether to enable the large language model detection function according to the given probability P%). The large model detection is triggered only when the sampling hits and the text length is not empty. If it does not hit, it directly downgrades to the first recognition result.

[0065] It is understood that if the random number is greater than the dynamic sampling factor, this embodiment will not include the corresponding paragraph as a sampled part. By determining whether to include the corresponding paragraph as a sampled part of the text based on whether the random number is less than the dynamic sampling factor, this embodiment can adjust the number of samples by adjusting the size of the dynamic sampling factor. Furthermore, this embodiment generates a random number for each text segment and samples different parts of the semantic text, thereby achieving text anomaly identification in different parts of the semantic text.

[0066] In summary, this embodiment performs two-level recognition on the text to be processed. In the first-level recognition, this embodiment obtains a first recognition result including semantic text, and after sampling the semantic text, it is input into a large language model. The large language model combines the semantics of the text to identify text anomalies. Since this embodiment performs partial sampling before inputting the text into the large language model, it avoids processing the entire text through the large language model, reducing the overhead of text anomaly recognition. Furthermore, since the partial sampling in this embodiment is performed on the recognized semantic text, it allows for anomaly recognition by combining the complete semantics of the text with the large language model, avoiding the identification of meaningful characters as garbled text, or the identification of garbled text with a few normal characters as normal, thereby improving the accuracy of text anomaly recognition. Therefore, this embodiment can balance the accuracy and cost of text anomaly recognition.

[0067] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30, the text anomaly handling method further includes steps S31-S32: Step S31: Perform a second-level recognition on the partial text using the large language model and output a second recognition result; Understandably, the first-level recognition can only judge text anomalies using fixed matching rules, which is prone to false positives and false negatives, thus affecting the accuracy of text anomaly recognition. Since the large language model is a model with semantic understanding capabilities, this embodiment uses the large language model in conjunction with the semantics of the sampled part of the text to judge text anomalies, thereby reducing false positives and false negatives and improving the accuracy of the large language model's judgment.

[0068] In its implementation, when performing second-level recognition on a portion of text using a large language model, this embodiment can combine the sampled text with preset second-level recognition prompts to construct the input to the large language model. The prompts are then used to judge various garbled text conditions, and the second recognition result is output. Alternatively, this embodiment can use the preliminary judgment result from the first-level recognition and language information as input to the large language model, allowing the model to use these as prior conditions to determine whether the first-level recognition result is valid and obtain the second recognition result.

[0069] Furthermore, in the second embodiment, the specific implementation of performing second-level recognition of the partial text using the large language model can also be: Based on the large language model and preset constraint rules, a second-level recognition is performed on the partial text. The preset constraint rules include at least one of the following: mapping garbled character processing rules, special character processing rules, local garbled character processing rules, and identifier processing rules.

[0070] It is understandable that the extracted text can be complex, containing mathematical formulas, engineering codes, special symbols, and tables. The logic for judging garbled text differs depending on the content. If a large language model is used for judgment without constraints, it is prone to errors. Therefore, this embodiment, based on the semantic understanding of the text by the large language model, uses constraint rules to perform anomaly recognition processing on different text contents, reducing the recognition errors of the large language model and improving the accuracy of text anomaly recognition.

[0071] In the specific implementation process, the processing rules for mapping garbled characters, special characters, local garbled characters, and identifiers in this embodiment are as follows: 1. Mapping garbled character handling rules: Force non-printable character or symbol sequences caused by incorrect mapping of Symbol / Wingdings fonts to be judged as garbled characters; 2. Special character processing rules: Force the determination that strings with specific domain contexts, such as engineering drawing numbers, invoices, certificate numbers, table data, and formulas, are free of garbled characters; 3. Local garbled text handling rules: Local character defects that account for a very small proportion and do not affect the overall semantic understanding of the paragraph are judged as not having garbled text; 4. Pure watermark and scanned document discrimination constraint: If the extracted content only contains repeated software watermarks or a very small number of fixed identifiers, it is judged as having garbled text, and it can also indicate that it may be a pure scanned document.

[0072] Furthermore, in this embodiment, during the process of calling the large language model for text anomaly recognition, there may be situations where the large language model cannot connect or the processing time is too long. To ensure text anomaly processing under different anomaly conditions, the specific implementation of performing a second-level recognition on the portion of text using the large language model to output the corresponding target anomaly result can also be: The large language model is used to perform a second-level recognition on the text. If the second-level recognition times out or is abnormal, the first recognition result is taken as the target abnormal result.

[0073] It is understandable that when the input text is long, the model load is high, or the network fluctuates, the large language model may not be able to return results within the specified time. Furthermore, in cases of model service rate limiting, interface errors, or network anomalies, it may be impossible to successfully connect to the large language model. Therefore, this embodiment, when a timeout or anomaly is detected in the large language model, directly uses the first recognition result as the target anomaly result. When the second-level recognition is normal, a high-precision recognition result is used; when anomalies occur, the first-level result is used as a fallback. Thus, even if the large language model service experiences a timeout or anomaly, it can still output the target anomaly result normally, improving the stability of text anomaly handling.

[0074] Furthermore, in this embodiment, when a connection to the large language model cannot be established, a reconnection attempt will be made. When the number of reconnections exceeds the reconnection threshold, the connection to the large language model is determined to be abnormal. After capturing the abnormality and logging it, the system automatically and seamlessly downgrades to the first recognition result. The reconnection threshold can be set according to requirements.

[0075] Step S32: Based on the second identification result, the first identification result is corrected in reverse, and the corrected first identification result is used as the target abnormal result.

[0076] It is understandable that the first-level identification is a matching method based on fixed rules, which may result in false positives and false negatives. Therefore, this embodiment corrects the false positives in the first identification result with the second identification result, and at the same time supplements the anomalies that were missed in the first level, thereby improving the accuracy of text anomaly identification.

[0077] It should be understood that in this embodiment, if the text to be processed does not have a second-level recognition result, the first-level recognition result is taken as the target abnormal result to avoid missing the text to be processed and to ensure that all texts to be processed are effectively recognized.

[0078] In a specific implementation, the way to correct the first identification result based on the second identification result can be to directly replace the first identification result with the second identification result, or to perform item-by-item verification on the first identification result based on the second identification result and supplement any missing abnormal items in the first identification result.

[0079] Furthermore, the second recognition result in this embodiment may only include the judgment result of whether the text is abnormal, or it may further include the language of the text. In order to achieve targeted processing of texts with different languages, the specific implementation of the method in this embodiment of reversing the first recognition result based on the second recognition result and using the corrected first recognition result as the target abnormal result may also be: Based on the second recognition result, if it is determined that the part of the text is not represented using a preset language, then the first recognition result is corrected to be non-garbled text.

[0080] It is understandable that the first and second level recognition in this embodiment are typically based on the character features, lexical structure, or garbled text patterns of a preset language for text anomaly identification. When the text actually contains a language other than the preset one, it is easily misjudged as garbled text due to the different prior knowledge distribution of these texts in the model, thus generating false alarms. Therefore, the large language model in this embodiment judges the language of the text. When the text is judged to be a minor language, it forces the corresponding text to be judged as not garbled text, thereby avoiding the mistaken judgment of minor languages ​​as garbled text and requiring additional subsequent processing.

[0081] In practice, the preset language can be set and modified to multiple languages ​​as needed. For example, the preset language can be set to English and Chinese, while the non-preset language can be any language other than the preset language. When the preset language is set to Chinese, the non-preset language can be Japanese, Korean, Vietnamese, etc.

[0082] In summary, this embodiment uses a large language model to perform semantic recognition on a portion of the sampled text, and then corrects the corresponding first recognition result using the second recognition result. Since the second recognition result is obtained by combining text semantics and constraint rules with the large language model, it can accurately judge different text contents. By correcting the first recognition result using the second recognition result and using the corrected first recognition result as the target anomaly result, the accuracy of text anomaly recognition can be improved.

[0083] Furthermore, in this embodiment, when the large language model times out or malfunctions, the first recognition result is directly used as the target anomaly result, thereby avoiding interruption of the text anomaly recognition process and improving the stability of text anomaly recognition. Moreover, this embodiment, by judging the language of the text, forces the text to be judged as non-garbled text when the language is a less common language, thereby preventing incorrect judgments from entering subsequent processing flows and reducing resource waste.

[0084] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S30, the text anomaly handling method further includes steps S100~S200: Step S100: If the text anomaly processing result is garbled text, then determine the text image corresponding to the text to be processed; Understandably, traditional text extraction pipelines typically interrupt or output corrupted text when encountering garbled text or extraction anomalies, failing to extract complete text from the original file. Therefore, this embodiment converts the text into an image format after identifying garbled text. After file format conversion, this embodiment can subsequently extract the text using different text extraction methods.

[0085] like Figure 4 The text anomaly handling process shown in this embodiment involves judging the anomaly of the text to be processed. After judging that there is garbled text in the text, the text is converted into a text image, and then the text is extracted from the text image, thereby realizing text extraction and improving the success rate of text extraction.

[0086] Furthermore, this embodiment can interrupt plain text extraction after determining that the extracted text is garbled, so as to stop using the damaged text extracted from the page, call the headless rendering engine, and render the text page into a high-resolution image without loss, thereby obtaining the corresponding text image.

[0087] Step S200: Perform secondary mapping reconstruction on the text image to obtain the reconstructed text, and then perform subsequent processing on the reconstructed text.

[0088] It is understandable that text images retain the glyphs, positions, and layout information of characters on the original carrier, but images differ from readable text in form and cannot be directly used for text processing. Therefore, this embodiment first maps the text image into an intermediate representation that can characterize visual features, and then reconstructs the intermediate representation into standard text, thereby achieving accurate extraction of text from the image.

[0089] In a specific implementation, this embodiment can extract the visual features of the text image through an encoder, decode the visual features to obtain the text through a decoder, or preprocess the text image, use a layout analysis algorithm to map the image into layout structure data, and then rearrange the layout structure data to obtain the reconstructed text.

[0090] Furthermore, in order to extract text from the image, the specific implementation method of performing secondary mapping and reconstruction on the text image to obtain the reconstructed text in this embodiment can also be: Determine the scaling factor and rotation angle of the page where the text image is located; Based on the scaling factor, rotation angle, and image edges of the text image, the position of the text corresponding to the text image on the page is determined, resulting in the reconstructed text.

[0091] It is understood that a text image contains information about the visual position, size, and angle of the text on the original page. Therefore, this embodiment accurately maps the text image back to its original page coordinate system by using scaling factors, rotation angles, and image edges to determine its corresponding text region. This allows for the direct extraction of readable text from the page's text layer, thus achieving text reconstruction.

[0092] In this implementation, the downstream OCR (Optical Character Recognition) engine is used for recognition, and the rendered image is submitted to the image OCR service to obtain the text line OCR results and the table OCR structure, respectively. In getParagraphsByLayout, the physical boundary BBox of the OCR item is extracted, and the text is reconstructed by combining the scaling factor and rotation angle of the page viewport while preventing text coordinate distortion.

[0093] During the OCR extraction process, this embodiment monitors the OCR extraction process through a standardized error code system. When OCR extraction fails, an error is reported and "4010 OCR parsing failure" is output, so that abnormalities in the OCR extraction process can be traced.

[0094] Furthermore, after text reconstruction, the text paragraphs introduced by OCR in this embodiment are aligned for boundaries and formatting. The code for boundary and formatting alignment is as follows: if (ocrItems?.length) { / / Check: If the OCR recognition result exists and is not an empty array, then perform subsequent processing to avoid performing meaningless operations on empty data. ocrItems.forEach((ocrItem) =>{ / / Iterate through all OCR recognition items and process them one by one. addParagraph(paragraphs, ocrItem); / / Adds the current OCR item as a paragraph object and stores it in the paragraphs array. const paragraph = paragraphs.at(-1); / / Get the paragraph object that was just added (i.e., the last element in the array). const height = paragraph.bottom - paragraph.top; / / Calculate the height of the paragraph: bottom coordinates minus top coordinates paragraph.type = 'text'; / / Marks the paragraph as "text" type for easier recognition and processing by the renderer. paragraph.textAlign = 'center'; / / Sets the horizontal alignment of the text within the paragraph to 'center'. paragraph.containerTop = paragraph.top + height 0.2; / / Set the top margin of the container: 20% below the top of the paragraph, leaving blank space above. paragraph.containerBottom = paragraph.bottom - height 0.2; / / Sets the bottom margin of the container: 20% of the height above the bottom of the paragraph, leaving blank space below. }); / / End iteration / / Termination condition check In this embodiment, the code described above converts each piece of text recognized by OCR into a corresponding paragraph object, and sets it to a uniform format with center alignment and 20% top and bottom margins, thereby ensuring that the format and attributes of the extracted text are correct.

[0095] In summary, this embodiment converts text into an image and extracts the text by extracting the text image. Since the text image contains information about the visual position, size, and angle of the text on the original page, this embodiment accurately maps the text image back to its original page coordinate system using scaling factors, rotation angles, and image edges to determine the corresponding text region. This allows for direct extraction of readable text from the page's text layer, achieving text reconstruction and improving the accuracy of text extraction. Furthermore, this application performs subsequent formatting and boundary alignment processing on the reconstructed text after reconstruction to ensure the correct format and attributes of the extracted text, further enhancing the text extraction effect.

[0096] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the text exception handling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0097] This application also provides a text anomaly handling device; please refer to... Figure 5 The text anomaly handling device includes: Text acquisition module 10 is used to acquire the text to be processed; It is understood that the text to be processed in this embodiment may come from various business scenarios, such as text content extracted from PDF, Word and other documents, text recognized by OCR from images, or plain text data directly pushed by upstream systems.

[0098] In this embodiment, the method of acquiring the text to be processed can be flexibly configured to adapt to different application scenarios and data scales. The specific data acquisition method can be to continuously acquire the text data generated by the upstream system in a streaming manner by accessing a distributed message queue or real-time data stream platform, or to read the text files to be processed in batches through a file system interface or object storage service. No limitation is made here.

[0099] In this embodiment, a standardized error code system is used during the data reading process. If file encryption or reading failure occurs, an error is reported and an error code such as "1012 File Encryption" or "4004 Read Failure" is output, so as to trace the abnormal file reading.

[0100] The first recognition module 20 is used to input the text to be processed into a preset pre-text processing model, perform first-level recognition on the text to be processed based on the pre-text processing model, and output a first recognition result, wherein the first recognition result includes at least the corresponding semantic text. Understandably, since the text to be processed can be text with complete semantics, or text without complete semantics such as mathematical formulas, special characters, engineering codes, and tables, if text without complete semantics is input into the large language model, the large language model will not be able to accurately determine whether the text is garbled.

[0101] Therefore, this embodiment performs preliminary identification on the text to be processed to obtain a first identification result containing at least the corresponding semantic text, so that the text input into the large language model is all text with complete semantics. Thus, text anomaly identification can be performed by combining the semantics of the text with the large language model, thereby improving the accuracy of text anomaly identification.

[0102] In a specific implementation, the first recognition result of this embodiment may contain only semantic text, or it may contain semantic text and a preliminary text anomaly judgment result. Based on the aforementioned pre-processing text model, this embodiment performs the first-level recognition of the text to be processed in the following ways: 1. Determine garbled characters and characters lacking complete semantic meaning in the text using preset matching rules, delete the garbled characters and characters lacking complete semantic meaning, and obtain a first recognition result containing the corresponding semantic text; 2. Based on the coordinate positions of the characters in the text to be processed, divide the text into multiple different text segments, determine whether each text segment has complete semantic meaning, and combine all text segments with complete semantic meaning to obtain the first recognition result.

[0103] The second recognition module 30 is used to sample a portion of the text from the semantic text and input the portion of the text into a preset large language model. The large language model is used to perform a second-level recognition on the portion of the text to output the corresponding target anomaly result.

[0104] The text anomaly processing device provided in this application, employing the text anomaly processing method in the above embodiments, can solve the technical problem of simultaneously achieving high accuracy and low cost in text anomaly recognition. Compared with the prior art, the beneficial effects of the text anomaly processing device provided in this application are the same as those of the text anomaly processing method provided in the above embodiments, and other technical features in the text anomaly processing device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0105] This application provides a text anomaly processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the text anomaly processing method in Embodiment 1 above.

[0106] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the text anomaly handling device of the embodiments of this application. The text anomaly handling device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The text exception handling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0107] like Figure 6 As shown, the text anomaly handling device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the text anomaly handling device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the text anomaly processing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show text anomaly processing devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0108] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0109] The text anomaly processing device provided in this application, employing the text anomaly processing method in the above embodiments, can solve the technical problem of being unable to simultaneously achieve high accuracy and low cost in text anomaly recognition. Compared with the prior art, the beneficial effects of the text anomaly processing device provided in this application are the same as those of the text anomaly processing method provided in the above embodiments, and other technical features of this text anomaly processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0110] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0112] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the text exception handling method in the above embodiments.

[0113] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0114] The aforementioned computer-readable storage medium may be included in the text exception handling device; or it may exist independently and not assembled into the text exception handling device.

[0115] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the text error handling device, cause the text error handling device to perform the aforementioned text error handling method.

[0116] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0119] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described text anomaly processing method. This solves the technical problem of simultaneously achieving high accuracy and low cost in text anomaly recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the text anomaly processing method provided in the above embodiments, and will not be repeated here.

[0120] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the text exception handling method described above.

[0121] The computer program product provided in this application can solve the technical problem of balancing the accuracy and cost of text anomaly recognition. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the text anomaly processing method provided in the above embodiments, and will not be repeated here.

[0122] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A text anomaly handling method, characterized in that, The method includes: Get the text to be processed; The text to be processed is input into a preset pre-text processing model. Based on the pre-text processing model, the text to be processed is subjected to a first-level recognition, and a first recognition result is output. The first recognition result includes at least the corresponding semantic text. A portion of the text is sampled from the semantic text and input into a preset large language model. The large language model performs a second-level recognition on the portion of the text to output the corresponding target anomaly result.

2. The method as described in claim 1, characterized in that, The step of sampling a portion of the text from the semantic text includes: Dynamically sample portions of text from the semantic text.

3. The method as described in claim 2, characterized in that, The step of dynamically sampling a portion of text from the semantic text includes: Generate random numbers for different paragraphs of the semantic text; Based on the comparison result between the random number and the preset dynamic sampling factor, it is determined whether to include the corresponding paragraph as part of the text to be sampled.

4. The method as described in claim 3, characterized in that, The step of determining whether to include a corresponding paragraph as part of the sampled text based on the comparison result of the random number and the preset dynamic sampling factor includes: If the random number is less than the dynamic sampling factor, then the corresponding paragraph will be used as the sampled part of the text.

5. The method as described in claim 1, characterized in that, The step of performing a second-level recognition on the portion of text using the large language model to output the corresponding target anomaly result includes: The large language model is used to perform a second-level recognition on the partial text, and a second recognition result is output. Based on the second identification result, the first identification result is corrected in reverse, and the corrected first identification result is taken as the target abnormal result.

6. The method as described in claim 5, characterized in that, The step of reversing the first identification result based on the second identification result and using the corrected first identification result as the target abnormal result includes: Based on the second recognition result, if it is determined that the part of the text is not represented using a preset language, then the first recognition result is corrected to be non-garbled text.

7. A text anomaly processing device, characterized in that, The device includes: The text acquisition module is used to acquire the text to be processed. The first recognition module is used to input the text to be processed into a preset pre-text processing model, perform first-level recognition on the text to be processed based on the pre-text processing model, and output a first recognition result, wherein the first recognition result includes at least the corresponding semantic text; The second recognition module is used to sample a portion of the text from the semantic text and input the portion of the text into a preset large language model. The large language model is used to perform a second-level recognition on the portion of the text to output the corresponding target anomaly result.

8. A text anomaly processing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the text anomaly handling method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the text exception handling method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the text exception handling method as described in any one of claims 1 to 6.

Images