Text processing method and device, electronic equipment, storage medium and program product
By segmenting text into independent semantic units and filtering out target semantic units, the problems of high false positive rate and high cost in traditional methods are solved, and the recognition accuracy and adaptability are improved without increasing computational overhead.
Patent Information
- Application Number
- CN202611378919.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-11-10
AI Technical Summary
Traditional rule engines suffer from high false positive rates and poor adaptability. Directly calling large language models to process the full text leads to a surge in interface call costs and strain on computing resources. Furthermore, they are prone to losing metadata when processing heterogeneous data, making it difficult to meet the requirements of financial compliance testing.
By segmenting the text to be recognized into multiple independent semantic units that express complete semantic meaning, the most discriminative target semantic units are selected, and only concise and semantically consistent fragments are encapsulated as model input, ensuring the semantic integrity of each information fragment and improving recognition accuracy without increasing model inference burden and token overhead.
While reducing the cost of model invocation, it improves the accuracy of recognizing complex sentence structures and implicit semantics, adapts to heterogeneous data, and ensures the recall and accuracy of the target task.
Smart Images

Figure CN123070128A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of text processing technology, and in particular to a text processing method, apparatus, electronic device, storage medium, and program product. Background Technology
[0002] With the digital transformation of the financial industry, the amount of unstructured documents generated daily, such as those for credit approvals and compliance reviews, is increasing dramatically, creating an urgent need for automated methods to achieve content recognition and risk assessment.
[0003] In related technologies, traditional rule engines are widely used for text matching and filtering, while large language models (LLMs) have begun to be applied to document processing due to their powerful semantic understanding capabilities, usually directly calling the entire text.
[0004] However, rule engines have a high false positive rate and poor adaptability; directly calling LLM to process the full text causes the interface call cost to increase linearly with the number of tokens, resulting in a shortage of computing resources. Furthermore, existing methods lack structure awareness when processing heterogeneous data such as tables and image recognition text, and are prone to losing metadata such as table headers, which damages semantic integrity and makes it difficult to meet the requirements of financial compliance testing. Summary of the Invention
[0005] This application provides a text processing method, apparatus, electronic device, storage medium, and program product. The method significantly compresses the amount of text input to the target recognition model by filtering and reconstructing semantic units, while ensuring semantic integrity and preventing the loss of key information. This reduces model calling costs and computing power consumption, while improving the accuracy of target task recognition and the ability to adapt to heterogeneous data.
[0006] In a first aspect, embodiments of this application provide a text processing method, including:
[0007] The text to be identified is obtained, and the text to be identified is semantically segmented to obtain multiple semantic units; wherein each semantic unit expresses an independent and complete semantic meaning;
[0008] At least one target semantic unit is selected from the plurality of semantic units; the target semantic unit is the selected semantic unit used as input content for constructing the target recognition model;
[0009] Based on the target semantic unit, input content that meets the input requirements of the target recognition model is generated; wherein, the input content is configured to trigger the target recognition model to perform target task recognition after being input into the target recognition model.
[0010] In one possible implementation, selecting at least one target semantic unit from the plurality of semantic units includes:
[0011] For each semantic unit, determine the expected token consumption and semantic point density of the semantic unit;
[0012] Based on the expected token consumption and semantic point density of each semantic unit, at least one target semantic unit is selected from the plurality of semantic units.
[0013] In one possible implementation, determining the expected token consumption of the semantic unit includes:
[0014] Determine the number of first-type characters and the number of second-type characters in the semantic unit; wherein, the first-type characters are characters whose token count as a single character conforms to a first preset rule, and the second-type characters are characters whose token count as a single character conforms to a second preset rule;
[0015] The expected token consumption of the semantic unit is determined based on the number of characters in the first type, the number of characters in the second type, and the preset token conversion rules.
[0016] In one possible implementation, the token conversion rule includes a preset calibration coefficient;
[0017] The method further includes:
[0018] The preset calibration coefficients are initialized to default values; wherein, the preset calibration coefficients correspond to the target recognition model, different target recognition models have their own preset calibration coefficients, and the default values are the default values corresponding to the target recognition model;
[0019] Every preset time period, the actual token consumption of the text to be identified in the target recognition model within the time period is obtained, as well as the expected token consumption calculated according to the preset token conversion rule;
[0020] If the difference between the actual token consumption and the expected token consumption is greater than a preset threshold, the preset calibration coefficient is updated.
[0021] If the difference is less than or equal to the preset threshold, the current value of the preset calibration coefficient remains unchanged.
[0022] In one possible implementation, updating the preset calibration coefficient includes:
[0023] Construct a calibration verification set; the calibration verification set contains multiple samples extracted from the processed text;
[0024] For each sample in the calibration verification set, calculate the base token consumption and the actual token consumption for each sample;
[0025] Sum the base token consumption and the actual token consumption for all the samples to obtain the total base token consumption and the total actual token consumption.
[0026] Candidate calibration coefficients are determined based on the base token consumption and the total actual token consumption.
[0027] If the difference between the ratio of the actual total token consumption to the basic total token consumption and the current preset calibration coefficient is greater than the preset convergence threshold, then the current preset calibration coefficient is processed according to the ratio using a smooth update strategy to obtain the updated preset calibration coefficient.
[0028] In one possible implementation, determining the semantic point density of the semantic unit includes:
[0029] Determine the number of entities, the negation structure marker value, and the proportion of stop words in the semantic unit;
[0030] Wherein, if the semantic unit is marked as containing a negation structure during the semantic segmentation stage, the negation structure marking value is a first preset value; otherwise, it is a second preset value; the stop word ratio is the ratio of the number of stop words in the semantic unit to the total number of words.
[0031] The semantic point density of the semantic unit is determined based on the number of entities, the negation structure marker value, the proportion of stop words, their respective preset weight coefficients, and the preset point density calculation rules.
[0032] In one possible implementation, determining the semantic point density of the semantic unit includes:
[0033] Obtain the initial semantic point density of the semantic unit; the initial semantic point density is calculated based on the number of entities, the negation structure marker value, and the proportion of stop words;
[0034] Detect whether the semantic unit contains a first type of preset context word and / or a second type of preset context word;
[0035] Based on the detection results, the initial semantic point density, and the preset semantic point density adjustment rules, the semantic point density of the semantic unit is calculated.
[0036] The preset semantic point density adjustment rule includes: if the semantic unit contains the first type of preset context words, then a first preset weight value is added to the initial semantic point density; if the semantic unit contains the second type of preset context words, then a second preset weight value is reduced to the initial semantic point density; the first preset weight value is positive and the second preset weight value is negative.
[0037] In one possible implementation, at least one target semantic unit is selected from the plurality of semantic units based on the expected token consumption and the semantic point density of each of the semantic units, including:
[0038] For each semantic unit, a cost-effectiveness score is calculated based on the expected token consumption and the semantic point density of the semantic unit.
[0039] The semantic units are sorted in descending order of their cost-effectiveness scores to obtain a sorted sequence of semantic units.
[0040] Starting from the first semantic unit in the semantic unit sequence, the expected token consumption of each semantic unit is accumulated sequentially to obtain the cumulative token consumption;
[0041] If the cumulative token consumption is less than the preset token consumption threshold, then the expected token consumption of the next semantic unit continues to be accumulated;
[0042] If the cumulative token consumption is greater than or equal to the preset token consumption threshold, then the accumulation stops, and all semantic units preceding the semantic unit involved in the current accumulation are determined as the target semantic unit.
[0043] In one possible implementation, obtaining the text to be recognized includes:
[0044] Obtain the original content to be identified; the original content includes at least one of text content, image content, and table content;
[0045] Based on the text extraction strategy corresponding to the content type of the original content, the original content is processed to extract text, thereby obtaining the text to be identified.
[0046] In one possible implementation, the original content includes table content;
[0047] The text extraction strategy based on the content type of the original content is used to perform text extraction processing on the original content to obtain the text to be identified, including:
[0048] Identify the header area and data area in the table content;
[0049] Based on the field identifiers in the header area, each data record in the data area is converted into a text statement in key-value pair form to obtain the text to be identified;
[0050] Specifically, when generating the text to be recognized, a priority retention flag is set for the text content corresponding to the header area; the priority retention flag is used to indicate that when filtering target semantic units from semantic units to generate the input content, the semantic units corresponding to the header area are retained first.
[0051] In one possible implementation, the original content includes text content;
[0052] The text extraction strategy based on the content type of the original content is used to perform text extraction processing on the original content to obtain the text to be identified, including:
[0053] Multiple text blocks are obtained from the text content, and the content features of each text block are calculated; the content features include at least one of hash features and similarity features.
[0054] Text blocks whose content features meet the preset repetition conditions are identified as redundant text blocks, and the redundant text blocks are filtered out to obtain the text to be identified after removing redundancy.
[0055] In one possible implementation, the original content includes image content;
[0056] The text extraction strategy based on the content type of the original content is used to perform text extraction processing on the original content to obtain the text to be identified, including:
[0057] Optical character recognition is performed on the image content to obtain initial recognized text and the page position information corresponding to the initial recognized text;
[0058] The text segmentation boundaries are determined based on the layout position information and / or the format control characters contained in the initial identified text;
[0059] The initial recognition text is segmented at the text segmentation boundary to obtain the text to be recognized.
[0060] In one possible implementation, the semantic segmentation of the text to be identified, resulting in multiple semantic units, includes:
[0061] Based on punctuation marks, the text to be identified is segmented into multiple initial text segments;
[0062] According to the preset semantic integrity rules, the initial text segments are merged to obtain the multiple semantic units.
[0063] In one possible implementation, the step of merging the initial text segments according to a preset semantic integrity rule to obtain the plurality of semantic units includes:
[0064] For each of the initial text segments, detect whether the initial text segment contains a negative word from a preset negative word library;
[0065] If the initial text segment contains a negative word, then determine whether the initial text segment and the subsequent initial text segment satisfy the conditions of subject agreement or subject omission;
[0066] If the conditions are met, the initial text segment is merged with the next initial text segment to obtain a semantic unit.
[0067] In one possible implementation, the step of merging the initial text segments according to a preset semantic integrity rule to obtain the plurality of semantic units includes:
[0068] For each initial text segment, it is detected whether the initial text segment meets the preset condition triggering condition; wherein, the preset condition triggering condition includes: the initial text segment begins with a word in the preset condition-guided lexicon and the semantics are incomplete, or the initial text segment ends with a comma and the semantics are incomplete;
[0069] If the initial text segment meets the preset trigger condition, then it is detected whether the next initial text segment contains a word from the preset result dictionary.
[0070] If included, the initial text segment is merged with the subsequent initial text segment to obtain a semantic unit.
[0071] In one possible implementation, the step of merging the initial text segments according to a preset semantic integrity rule to obtain the plurality of semantic units includes:
[0072] For each of the initial text segments, detect whether the initial text segment contains words from a preset enumerated guide dictionary;
[0073] If the initial text segment contains words from the preset enumeration guide vocabulary, then the initial text segment is determined as the enumeration starting segment;
[0074] Starting from the enumeration start segment, sequentially check whether the next initial text segment meets the preset enumeration end condition;
[0075] The enumeration starting segment and all subsequent initial text segments up to and including the preset enumeration ending condition are merged to obtain a semantic unit.
[0076] In one possible implementation, generating input content that satisfies the input requirements of the target recognition model based on the target semantic unit includes:
[0077] Obtain the original semantic position of each target semantic unit in the text to be identified;
[0078] According to the order of the original semantic position representations, the target semantic units are sorted to obtain a sorted sequence of target semantic units;
[0079] Using a preset connector, each target semantic unit in the target semantic unit sequence is connected sequentially to obtain the connected text;
[0080] The input content is obtained by encapsulating the connection text using a preset prompt template.
[0081] Secondly, embodiments of this application provide a text processing apparatus, including:
[0082] The semantic unit acquisition module is used to acquire the text to be identified, perform semantic segmentation on the text to be identified, and obtain multiple semantic units; wherein, each semantic unit expresses an independent and complete semantic meaning;
[0083] A semantic unit filtering module is used to filter out at least one target semantic unit from the plurality of semantic units;
[0084] The input content generation module is used to generate target input content based on the target semantic unit; wherein the target input content is content that meets the input requirements of the target recognition model and is intended for target task recognition by the target recognition model.
[0085] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0086] The memory stores computer-executed instructions;
[0087] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0088] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0089] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0090] The text processing method, apparatus, electronic device, storage medium, and program product provided in this application segment the text to be identified into multiple semantic units, each expressing independent and complete semantics. This ensures the semantic integrity of each information fragment from the source, preventing semantically incomplete fragments from being generated during subsequent screening. Based on this, at least one target semantic unit most valuable for text content recognition is selected from all semantic units, excluding a large amount of redundant content irrelevant to risk assessment. Furthermore, only the filtered, concise, and semantically self-consistent fragments are encapsulated as input content and fed into the target recognition model. This results in highly condensed information density in the input received by the model, while maintaining complete semantic logic. Thus, without increasing the model's inference burden or generating additional token overhead, the recall and accuracy of the target task recognition are guaranteed, reducing invocation costs and improving recognition reliability. Attached Figure Description
[0091] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0092] Figure 1 An application scenario diagram of the text processing method provided in this application;
[0093] Figure 2 A flowchart illustrating a text processing method provided in this application;
[0094] Figure 3 A flowchart illustrating another text processing method provided in an embodiment of this application;
[0095] Figure 4 A schematic diagram of the text processing apparatus provided in this application;
[0096] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.
[0097] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0098] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0099] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0100] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of the relevant data shall comply with relevant laws, regulations and standards, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.
[0101] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0102] With the digital transformation of the financial industry, the amount of unstructured documents generated daily, such as those for credit approvals and compliance reviews, is increasing dramatically, creating an urgent need for automated methods to achieve content recognition and risk assessment.
[0103] In related technologies, traditional rule engines are widely used for text matching and filtering, while large language models (LLMs) have begun to be applied to document processing due to their powerful semantic understanding capabilities, usually directly calling the entire text.
[0104] However, rule engines have a high misjudgment rate and poor adaptability; directly calling LLM to process the full text causes the model call cost to increase linearly with the number of tokens, resulting in a shortage of computing resources. Furthermore, existing methods lack structure awareness when processing heterogeneous data such as tables and image recognition text, easily losing metadata such as table headers, destroying semantic integrity, and making it difficult to meet the requirements of financial compliance detection.
[0105] To address the aforementioned technical issues—namely, the surge in calling costs and computational strain caused by directly feeding the full text of financial documents into a large language model, and the tendency of traditional truncation or rule-based filtering to fragment semantic information when processing heterogeneous data such as tables and image text—this application provides a text processing method. This method pre-segments the text to be recognized into multiple independent semantic units that express complete semantic meaning, achieving self-containment for each information fragment and preventing semantic fragmentation caused by segmentation. Based on this, the most discriminative target semantic units are selected, and only concise and semantically consistent fragments are encapsulated as model input. Since each semantic unit itself contains all the logic required for target content recognition, the model can accurately recognize the content without retrieving the entire text. This proportionally compresses the input length, significantly reduces computational overhead and model calling costs, while improving the accuracy of recognizing complex sentence structures and implicit semantics.
[0106] The text processing method provided in this application is applicable to various business scenarios that require target task recognition of massive amounts of text.
[0107] For example, in financial compliance review scenarios, unstructured documents such as credit contracts and regulatory reports can be preprocessed using this method and then fed into a large language model, which can accurately identify sensitive clauses and risk information while reducing API call costs.
[0108] Furthermore, the above methods can also be applied to scenarios requiring long text processing, such as text classification, sentiment analysis, and intelligent question answering. In summary, any text content processing scenario that needs to balance the accuracy of large model recognition with computational overhead falls within the application scenarios of the text processing technology solution in this application.
[0109] The following is combined with Figure 1 The application scenarios applicable to the embodiments of this application are described below. Figure 1 A diagram illustrating an application scenario of the text processing method provided in this application. See also... Figure 1 Taking the compliance review of financial credit approval materials as an example, this scenario involves user terminals (used to upload credit documents to be reviewed) and electronic devices (used to perform text preprocessing and call large language models for target task recognition).
[0110] Based on this, the technical solution proposed in this application includes the following steps in this scenario:
[0111] Users upload credit application materials, including forms, images, or text, to electronic devices via their user terminals;
[0112] The electronic device acquires the text to be recognized, performs semantic segmentation on the text to be recognized, and obtains multiple semantic units, each of which expresses an independent and complete semantic meaning;
[0113] The electronic device selects at least one target semantic unit from multiple semantic units;
[0114] Electronic devices generate concise input content based on target semantic units, send the input content to a target recognition model deployed on electronic devices or in the cloud for target task recognition (e.g., sensitive content recognition), and return the recognition results to the user terminal (e.g., prompting whether there are risk clauses or sensitive information).
[0115] Through the above steps, semantic simplification and reconstruction of financial credit documents are achieved. While preserving the complete semantics, the amount of text input to the large language model is significantly reduced, thereby reducing the model calling cost and improving the accuracy of identifying sensitive clauses and risk information.
[0116] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0117] Figure 2 This is a flowchart illustrating a text processing method provided in this application. Figure 2 As shown, the method includes:
[0118] S201. Obtain the text to be recognized, perform semantic segmentation on the text to be recognized, and obtain multiple semantic units; wherein, each semantic unit expresses an independent and complete semantic.
[0119] In this embodiment, the text to be identified can be text content obtained based on different types of original content. In one optional implementation, the text to be identified is original text directly provided by the user or system, such as a plain text contract, the body of an email, or a record of an instant message. In another optional implementation, the text to be identified is text extracted from tabular or image data after preprocessing.
[0120] After acquiring the text to be recognized, semantic segmentation is performed to obtain multiple semantic units. It should be understood that the semantic segmentation in this embodiment is not a simple mechanical breakup based on punctuation marks, but rather a granular division with the core objective of expressing an independent and complete semantic meaning. That is, each semantic unit obtained through segmentation is the smallest semantic fragment in the text to be recognized that independently expresses complete information without relying on other segments.
[0121] In one implementation of semantic segmentation of the text to be identified, a dependency parser in natural language processing can be used to directly parse the syntactic structure of the text to be identified, identify the dependency relationships between the components, and output the syntactic subtree with a complete subject-predicate structure or independent semantics as a semantic unit.
[0122] In another approach, the text to be recognized can be input into a pre-trained lightweight semantic segmentation model. After being trained on a dataset labeled with semantic unit boundaries, the model can directly output a sequence of text fragments with semantic unit labels, where each text fragment is a semantic unit.
[0123] Of course, other segmentation methods that can divide text into independent and complete semantic segments can also be used, and this application does not specifically limit this.
[0124] Through the semantic segmentation process described above, the text to be identified is transformed into a set of multiple semantic units. Each semantic unit not only carries its own complete semantic content but also includes its position information in the original text and semantic feature markers, including negation markers. This method of reconstructing long texts into fine-grained, semantically self-contained units allows subsequent filtering steps to make precise selections based on the density of semantic key points. At the same time, it ensures that the content input to the target recognition model is highly concise while still fully retaining all the context required for target content recognition, providing a reliable guarantee for achieving both low cost and high accuracy in recognition.
[0125] S202. Select at least one target semantic unit from multiple semantic units.
[0126] In this embodiment, the target semantic unit is the selected semantic unit of the input content used to construct the target recognition model.
[0127] After obtaining multiple semantic units, at least one target semantic unit with discriminative value for the target task is selected from these semantic units in order to compress the input size to the greatest extent while ensuring the recognition accuracy.
[0128] In one optional implementation, the selection process can be based on the expected token consumption and semantic point density of each semantic unit. The expected token consumption measures the number of tokens required to input the semantic unit into the target recognition model. The semantic point density measures the density of semantic features related to the target content within the semantic unit. Based on these two metrics, a cost-effectiveness score is calculated for each semantic unit. These units are then sorted from highest to lowest cost-effectiveness score, and, provided that the cost-effectiveness does not exceed a preset token consumption threshold, the top-ranked semantic units are selected as the target semantic units.
[0129] Since the screening process has fully considered the balance between token budget constraints and semantic point density, the screening results can effectively cover the core target information in the text to be identified, and can also control the content size of the input target recognition model within a reasonable range, thereby reducing the cost of model invocation.
[0130] S203. Based on the target semantic unit, generate input content that meets the input requirements of the target recognition model; wherein, the target input content is content that meets the input requirements of the target recognition model and is used by the target recognition model to perform target task recognition.
[0131] In this embodiment, after selecting at least one target semantic unit from multiple semantic units, these target semantic units are transformed into input content that meets the input requirements of the target recognition model, so that the target recognition model can directly process the input content and realize the processing task of quickly performing target task recognition.
[0132] In one optional implementation, the target semantic units can be concatenated according to their order of appearance in the original text to be identified, forming a semantically coherent and highly concise text sequence, which can then be directly used as input. Since each target semantic unit has been segmented into independent fragments that express complete semantics, even if only the filtered semantic units are retained and recombined, the resulting text sequence can still maintain the semantic self-containment of each information point, and will not lead to semantic breaks or ambiguities due to the omission of redundant or irrelevant parts of the original text.
[0133] In another implementation, each target semantic unit can be constructed as an independent input item and fed into the target recognition model in batch input for recognition one by one, thereby obtaining fine-grained recognition results for each target semantic unit.
[0134] In another implementation, preset instruction text or contextual guidance information can be added to the concatenation or encapsulation of each target semantic unit to generate input content that better conforms to the interaction format of the target recognition model. Furthermore, other methods can also be used, and this embodiment is not limited to any of them.
[0135] After the generated input content is fed into the target recognition model, the model uses its semantic understanding capabilities to identify the target within the input content and outputs the corresponding recognition results. It should be understood that the target recognition mentioned in this embodiment can include various recognition tasks, including but not limited to intent recognition and domain segmentation in text classification, sentiment tendency recognition in sentiment analysis, and context matching and retrieval enhancement in intelligent question answering. Furthermore, the model can also use its semantic understanding capabilities to specifically determine whether the input content contains predefined sensitive data types and output corresponding recognition results. These results may include sensitive category labels, risk levels, and the location range of sensitive content in the original text, for use by downstream alarm, blocking, or desensitization processing modules.
[0136] The text processing method provided in this application segmentes the text to be identified into multiple semantic units, each expressing independent and complete semantic meaning. This ensures the semantic integrity of each information fragment from the source, preventing semantically incomplete fragments from being generated during subsequent filtering. Based on this, at least one target semantic unit most valuable for target content identification is selected from all semantic units, excluding a large amount of redundant content irrelevant to risk assessment. Furthermore, only the filtered, concise, and semantically consistent fragments are encapsulated as input content and fed into the target recognition model. This results in highly condensed information density in the input received by the model, while maintaining complete semantic logic. Thus, without increasing the model's inference burden or generating additional token overhead, the recall and accuracy of the target task recognition are guaranteed, reducing invocation costs and improving recognition reliability.
[0137] Figure 3 This is a flowchart illustrating another text processing method provided in an embodiment of this application. Next, we will combine... Figure 3 The technical principles of the semantic comparison technology for the query script provided in this application will be further explained. It should be noted that the following description is merely an exemplary implementation of the technical solution of this application and does not constitute any limitation on the technical solution of this application.
[0138] In this embodiment, an optional implementation method for obtaining the text to be identified may include: obtaining the original content to be identified; the original content includes at least one of text content, image content, and table content; and performing text extraction processing on the original content based on the text extraction strategy corresponding to the content type to which the original content belongs, to obtain the text to be identified.
[0139] In this embodiment, the original content to be identified is obtained. The original content refers to various electronic documents or data carriers that may contain target information, and its content type may include at least one of text content, image content, and table content.
[0140] For example, text content can be plain text terms in a credit contract, the body of a compliance review report, or the text content of an email; image content can be scanned copies of paper documents, photographs of documents taken with a mobile phone, or screenshots containing text; and table content can be financial statements, regulatory reports, or Excel spreadsheets embedded in documents.
[0141] The original content can be obtained through various means such as user uploads, system interface imports, and database retrieval, and this embodiment does not limit this.
[0142] After obtaining the original content, it is necessary to perform text extraction processing based on the content type of the original content, using the corresponding text extraction strategy to obtain the text to be recognized in a uniform format. Since the way text information is carried differs fundamentally in different types of original content, differentiated extraction methods are required.
[0143] For example, when the original content is an image, text can be extracted from the image using optical character recognition (OCR) technology, converting the text within the image into an editable text sequence. When the original content is a table, the table structure can be identified using a document parsing interface or regular expressions, mapping the header fields and data records row by row into key-value pairs of text statements. For instance, the table row "Date: 2023-01-01, Amount: 1 million" can be converted into the natural language sentence "Date: January 1, 2023, Amount: 1 million.", thus obtaining the text to be recognized that can be used for semantic segmentation. This embodiment does not limit the method of obtaining the text to be recognized.
[0144] By using the above methods, various heterogeneous data sources can be uniformly converted into text to be recognized in a standardized format. This solves the problem that subsequent semantic segmentation cannot be uniformly processed due to the diversity of data sources and heterogeneous formats in real-world scenarios such as finance. It also improves the universality and reliability of the text processing workflow under multi-source heterogeneous input conditions.
[0145] Since the way text information is carried differs fundamentally in different content types, this embodiment adopts appropriate preprocessing strategies for text extraction for table content, image content and text content respectively, which will be explained one by one below.
[0146] In this embodiment, when the original content includes table content, a processing strategy for converting the original content into text to be recognized may include: recognizing the header area and data area in the table content; converting each data record in the data area into a key-value pair text statement according to the field identifier in the header area to obtain the text to be recognized; wherein, when generating the text to be recognized, a priority retention flag is set for the text content corresponding to the header area; the priority retention flag is used to indicate that when filtering target semantic units from semantic units to generate input content, the semantic units corresponding to the header area are retained first.
[0147] In this embodiment, since the table data has a structured row and column layout, the header fields define the semantic attributes of the data, and the data rows are the specific semantic content under the semantic attributes of the header. Therefore, in the conversion process, it is necessary not only to complete the morphological change from table to text, but also to fully preserve the semantic relationship between the header and the data, so as to avoid the target recognition model being unable to accurately understand the meaning of the data due to information loss in the subsequent filtering process.
[0148] In one implementation, when extracting text from table content, the header area and data area within the table content can be identified first. The header area is typically the first row or the first few rows of the table, containing the names of each column field, such as "Date," "Amount," "Counterparty," and "Contract Number." The data area comprises the rows outside the header area, with each row constituting an independent data record. Based on this, the header area and data area can be identified separately using the table's structural features.
[0149] For example, for HTML tables, the header area can be located by distinguishing between tags. For spreadsheet files such as Excel, the header area can be determined by reading the first row or a preset range of header rows. For tables in PDFs or scanned documents, the header area can be determined by combining visual layout features (such as bold first row, center alignment, background fill, and other formatting attributes) and text coordinate position information.
[0150] Based on this, each field name in the header area can be used as the "key" and the content of the corresponding column in the data record can be used as the "value" to generate a text statement in natural language form according to the preset splicing template.
[0151] For example, assuming the table header fields are "Date", "Transaction Amount", and "Counterpartner" in sequence, and a data record contains "January 1, 2023", "1 million yuan", and "Company A", the converted text statement can be "Date: January 1, 2023, Transaction Amount: 1 million yuan, Counterpartner: Company A". This key-value pair format of text statement explicitly expresses the semantic relationships between table columns into natural language, making each data record a self-contained semantic fragment that can be understood independently even outside the two-dimensional layout of the table.
[0152] Furthermore, during the generation of the text to be recognized, a priority retention flag can be set for the text content corresponding to the header area. This priority retention flag indicates that when filtering target semantic units from semantic units to generate input content, the semantic units corresponding to the header area should be retained first. For example, after converting the header content into a text statement, a priority flag field can be added to the statement, marking its priority as "high". By setting the priority retention flag, it can be ensured that the data structure skeleton represented by the header will not be truncated due to insufficient resources during subsequent filtering, so that the content input to the target recognition model always contains the semantic reference of field names. This avoids the "number without name" problem caused by only retaining data values and losing field semantics, ensuring the accurate recognition of tabular content by the target recognition model.
[0153] Through the above processing strategy, the table content is converted into linear text to be recognized that retains the semantic reference of the table header and has a priority retention mechanism, providing a well-structured input foundation for subsequent semantic segmentation and filtering.
[0154] In this embodiment, when the original content includes image content, a processing strategy for converting the original content into text to be recognized may include: performing optical character recognition on the image content to obtain initial recognized text and page position information corresponding to the initial recognized text; determining the text segmentation boundary based on the page position information and / or the format control characters contained in the initial recognized text; and segmenting the initial recognized text at the text segmentation boundary to obtain the text to be recognized.
[0155] In this embodiment, the image content typically includes scanned copies of paper documents, photographs of documents taken with a mobile phone, screenshots containing text, etc. The text information exists in pixel form and needs to be extracted before it can be used in the text processing workflow. Furthermore, the text generated after image text extraction often exhibits fragmented characteristics and lacks punctuation, which, if directly used as the text to be recognized, will severely affect the accuracy of subsequent semantic segmentation. Therefore, this embodiment, based on text extraction, further performs layout-aware segmentation processing to form text to be recognized with reasonable boundaries.
[0156] In one implementation, during the process of extracting text from image content, Optical Character Recognition (OCR) can be performed on the image content to obtain initial recognized text and the corresponding layout position information. In this process, the OCR engine typically outputs not only the recognition result of each character but also the layout position information of each character or text line in the original image. This layout position information may include the coordinate information of the character or text line (such as the horizontal and vertical coordinates of the top left and bottom right corners), line spacing information, indentation offset information, etc. For example, for a scanned image containing multiple segments of text, the OCR engine can output the text content of each segment, as well as the layout position information of that segment in the image, such as the starting Y-axis coordinate, ending Y-axis coordinate, and starting X-axis coordinate.
[0157] In addition, since the initial recognized text generated by OCR often lacks punctuation marks, it is impossible to directly perform semantic segmentation based on regular punctuation marks such as periods and semicolons. Therefore, it is necessary to use layout information and format control characters to simulate paragraph boundary judgment.
[0158] In determining text segmentation boundaries, one approach is to utilize formatting control characters found in the initially identified text. Formatting control characters refer to non-printing characters in the text sequence used to control layout formatting, such as consecutive line breaks, tabs, spaces, and indentation. When consecutive line breaks (e.g., two consecutive line breaks \n\n, indicating significant paragraph spacing in the original image) or significant indentation changes (e.g., multiple consecutive spaces or tabs at the beginning of a line, indicating first-line indentation in the original image) are detected, that location can be designated as a text segmentation boundary.
[0159] Another approach is to utilize page layout information to determine text segmentation boundaries. It should be understood that page layout information reflects the spatial arrangement of text within the original image. When the Y-axis coordinate displacement between adjacent text lines exceeds a preset threshold, it indicates the presence of significant paragraph spacing or region separation in the original image. Based on this, the location can be designated as a text segmentation boundary. Similarly, when the X-axis starting coordinate of a text line significantly shifts relative to the previous text line, it indicates indentation or segmentation, which can also be designated as a text segmentation boundary. In practical applications, the Y-axis displacement threshold can be set to several multiples of the normal line spacing (e.g., 1.5 or 2 times) to accommodate images with different layout densities.
[0160] After determining the text segmentation boundaries, the initial text to be recognized is segmented at these boundaries to obtain the text to be recognized. For example, paragraph marks are inserted or segmentation is performed at each text segmentation boundary, so that the originally fragmented, punctuation-free continuous text is divided into multiple text segments with reasonable boundaries. These text segments are the text to be recognized.
[0161] Through the above implementation method, the text information in the image content was successfully extracted, and the paragraph structure of the original text was restored with the help of the layout position information and format control characters. This effectively solved the problem of text fragmentation caused by the lack of punctuation marks, and provided the text to be recognized with clear boundaries and semantic coherence for subsequent semantic segmentation processing.
[0162] In this embodiment, when the original content includes text content, a processing strategy for converting the original content into text to be recognized may include: obtaining multiple text blocks in the text content, calculating the content features of each text block; the content features include at least one of hash features and similarity features; determining text blocks whose content features meet preset repetition conditions as redundant text blocks, and filtering out redundant text blocks to obtain the text to be recognized after removing redundancy.
[0163] In some scenarios, the text content is often not directly identifiable, but may contain a large amount of recurring redundant information, such as headers and footers, dynamic watermarks, disclaimers, and duplicate announcements. This redundant information appears repeatedly across multiple pages of a document, not only increasing the amount of text to be processed but also potentially crowding out the limited token budget during the semantic unit filtering stage, causing truly valuable substantive content to be excluded, thus affecting the accuracy of the target task's recognition. Therefore, when acquiring content, redundancy detection and filtering should be performed on the acquired text to remove duplicate information at the source and improve the information density of the text to be recognized.
[0164] Specifically, multiple text blocks are obtained from the text content. Here, a text block refers to a unit of text that can be divided into segments within the text content. The granularity of text block division can be set according to actual needs. For example, the text content can be divided into multiple text blocks according to paragraph marks, using natural paragraphs as units; text blocks can also be obtained by sliding the window with a fixed number of characters or sentences; or the division can be based on document structure features (such as chapter titles, page breaks, etc.).
[0165] After obtaining the text blocks, the content features of each text block are calculated. These content features are used to characterize the uniqueness or similarity of the text block content and may include at least one of hash features and similarity features.
[0166] Here, hash characteristics refer to fixed-length digest values obtained by applying a preset hash algorithm to the text content of a text block. For example, the MD5 or SHA-256 algorithm can be used to calculate a unique hash value for the text string of the text block. If the hash values of two text blocks are completely identical, it indicates that the text content of the two text blocks is exactly the same, and they can be determined as exact duplicates.
[0167] Similarity features are quantitative indicators used to measure the degree of content similarity between two text blocks. For example, the SimHash algorithm can be used to calculate the local target hash value of a text block, and the similarity can be determined by comparing the Hamming distance between the SimHash values of two text blocks. Alternatively, the Jaccard coefficient can be used, where the text block is segmented into words, and the ratio of the intersection to the union of the two word sets is calculated as the similarity. Other similarity measures, such as cosine similarity and edit distance, can also be used.
[0168] After obtaining the content features, text blocks whose content features satisfy a preset repetition condition are identified as redundant text blocks. These redundant text blocks are then filtered out to obtain the text to be identified after redundancy removal. The preset repetition condition can be set according to the type of content feature used. For example, when using hash features for redundancy detection, the preset repetition condition is: the hash value of the current text block is completely identical to the hash value of any processed text block. Once a hash value duplication is detected, the current text block is marked as a redundant text block. When using similarity features for redundancy detection, the preset repetition condition is: the similarity value between the current text block and any processed text block exceeds a preset similarity threshold. For example, the similarity threshold can be set to 0.95 or 0.9. When the similarity between two text blocks is greater than or equal to this threshold, they are determined to be highly similar in content, and the current text block is marked as a redundant text block.
[0169] Text blocks marked as redundant are directly removed from the text content, retaining only the first occurrence or unique text blocks. The remaining text blocks after filtering constitute the text to be identified after redundancy removal.
[0170] In this way, high-frequency repetitive information such as headers and footers, watermarks, and duplicate announcements in the text content are effectively identified and filtered out during the preprocessing stage. On the one hand, this reduces the overall length of the text to be identified and lowers the computational cost of subsequent semantic segmentation and filtering. On the other hand, it also avoids redundant information crowding out the limited token budget, ensuring that subsequent filtering steps can concentrate token resources on substantive content with unique semantic value, thereby guaranteeing the coverage and accuracy of the target task identification.
[0171] After obtaining the text to be recognized, it needs to be semantically segmented to obtain multiple semantic units. In one embodiment, the semantic segmentation process may include: breaking the text to be recognized into sentences according to punctuation marks to obtain multiple initial text segments; and merging the initial text segments according to preset semantic integrity rules to obtain multiple semantic units.
[0172] Specifically, preset sentence termination symbols (such as periods, question marks, exclamation marks, etc.) can be used as segmentation criteria, and the text to be identified can be divided into sentence-level fragment sequences through regular expression matching.
[0173] For example, the regular expression `re.split(r'(?<=[.!?])\s+', text)` can be used to perform this sentence segmentation operation. That is, this expression segments the text after each period, question mark, or exclamation mark, following a whitespace character, breaking the text down into several initial text segments and retaining the terminating character at the end of each segment, thus obtaining a list of initial text segments. In addition to the aforementioned terminating characters, semicolons, line breaks, and other symbols can also be included in the segmentation criteria based on the actual formatting characteristics of the text to obtain more fine-grained initial fragments.
[0174] Since relying solely on punctuation for sentence segmentation often fails to guarantee that each initial text segment expresses independent and complete semantics—for example, negation structures may be incorrectly separated by periods, conditional clauses and result clauses may be divided into different paragraphs, and enumeration items may be broken up—it is necessary to further perform semantic integrity merging processing.
[0175] During the merging process, adjacent initial text segments can be judged according to preset semantic integrity rules. Multiple initial text segments that are separated by punctuation but are semantically interdependent and together constitute a complete information point can be merged into a single semantic unit. The preset semantic integrity rules can be configured to identify and repair various typical incomplete semantic patterns, such as: a negation integrity rule based on negation word detection and subject association judgment to ensure that the negation word and its semantic object are not separated; a conditional integrity rule based on the identification of the pairing of conditional guide words and result words to maintain the complete logical connection of conditional sentences; and an enumeration integrity rule based on enumeration guide words and termination condition judgment to aggregate scattered enumeration items into a whole semantic unit.
[0176] After the above merging process, multiple semantic units are obtained, each with its own complete semantics. This effectively overcomes the semantic fragmentation problem caused by simple punctuation segmentation and provides high-quality processing objects for subsequent semantic unit selection and model input construction.
[0177] Furthermore, in other implementations, the semantic segmentation process described above can also be implemented using the following other methods. For example, segmentation can be based on a dependency parser, that is, using a dependency parser in natural language processing to parse the syntactic structure of the text to be identified, identify the dependency relationships between the components, and directly output the syntactic subtree with a complete subject-predicate structure or independent semantic function as a semantic unit. This method does not require the two-stage processing of first segmenting and then merging sentences; semantic boundaries can be located in a single analysis. Another example is segmentation based on a pre-trained semantic segmentation model. That is, the text to be identified is input into a pre-trained lightweight semantic segmentation model. After being trained on a dataset labeled with semantic unit boundaries, the model can directly output a sequence of text fragments with semantic unit labels, where each text fragment is a semantic unit. It should be understood that in practical applications, the appropriate method can be flexibly selected according to accuracy requirements and computing resources; this embodiment does not impose specific limitations on this.
[0178] In the above implementation process, one implementation method for merging each initial text segment according to the preset semantic integrity rules to obtain multiple semantic units may include: for each initial text segment, detecting whether the initial text segment contains a negative word from the preset negative word library; if the initial text segment contains a negative word, determining whether the initial text segment and the subsequent initial text segment meet the conditions of subject consistency or subject omission; if they meet the conditions, merging the initial text segment and the subsequent initial text segment to obtain a semantic unit.
[0179] In everyday textual expression, negative words (such as "not," "not," "without," "non-," "prohibited," etc.) are often grammatically closely related to the content they negate. However, in the initial sentence segmentation stage based on punctuation, the clause containing the negative word and its semantically governed subsequent content may be separated into two independent initial text segments due to punctuation. For example, "Our company has not discovered" and "any violations" are separated into different initial text segments by periods or commas. If this is not addressed, the former, as an isolated semantic unit, will express a semantic tendency completely opposite to the original text, leading to misjudgment by the target recognition model during subsequent target task identification. To avoid the above problems, this implementation method can use the negation integrity rule to realize cross-segment semantic association judgment during merging, thereby repairing the incorrectly segmented negative structures.
[0180] Specifically, for each initial text segment, it is checked whether the initial text segment contains a negative word from a preset negative word library. The preset negative word library can contain common negative words, and can be configured, for example, to contain a set of words such as "not", "no", "none", "non", "prohibited", "without", "not yet", and "never". This word library can be expanded or trimmed according to actual application needs.
[0181] If an initial text segment is detected to contain a negative word, it is further determined whether this initial text segment and its successor satisfy the conditions of subject agreement or subject omission. Subject agreement can be determined through dependency parsing or subject extraction algorithms, identifying whether the predicate subject of the preceding segment and the subject or implicit subject of the following segment refer to the same entity. Subject omission can be determined by considering Chinese language conventions; when the subsequent initial text segment is a subjectless statement or begins with a predicate, its subject is assumed to be omitted from the preceding sentence, thus agreeing with the subject of the preceding segment. If the above conditions of subject agreement or subject omission are met, it indicates that the two initial text segments semantically belong to the same negative structure, and they are then merged into a single semantic unit.
[0182] For example, in the initial text segment "Our company has not found any violations" and "any violations", the detection shows that the first segment contains the negative word "not" and the second segment is an object phrase without a subject, which satisfies the condition of subject omission. After merging the two, the semantic unit "Our company has not found any violations" is obtained, which completely restores the negative semantics of the original text.
[0183] By merging the negation integrity rules, the semantic fragmentation problem caused by punctuation breaks is effectively prevented, which separates the negation word from its object. This ensures that the resulting semantic units can accurately and completely carry the semantic information of the original text, providing a reliable foundation for the accurate identification of the target content.
[0184] In the above implementation process, another implementation method for merging each initial text segment according to the preset semantic integrity rules to obtain multiple semantic units may include: for each initial text segment, detecting whether the initial text segment meets the preset condition triggering condition; wherein, the preset condition triggering condition includes: the initial text segment starts with a word in the preset condition guiding dictionary and is semantically incomplete, or the initial text segment ends with a comma and is semantically incomplete; if the initial text segment meets the preset condition triggering condition, then detecting whether the next initial text segment contains a word in the preset result dictionary; if it contains a word, then merging the initial text segment with the next initial text segment to obtain a semantic unit.
[0185] In formal texts such as contract terms and compliance review reports, conditional sentences typically present a complex logical structure of a conditional clause plus a result clause, such as "If one party violates the confidentiality obligations under this agreement, it shall bear the corresponding liability for breach of contract." However, in the initial sentence segmentation stage based on punctuation, the conditional clause and its corresponding result clause are often separated into two independent initial text segments by commas, semicolons, or periods, forming two isolated fragments: "If one party violates the confidentiality obligations under this agreement" and "it shall bear the corresponding liability for breach of contract." If such cases are not merged, the former becomes a dangling premise without a conclusion, and the latter becomes an abrupt conclusion without a premise. The semantic units lose their logical self-containment, directly affecting the subsequent target recognition model's accurate judgment of the nature and risk level of the clauses. Based on this, the conditional integrity rule provided in this embodiment merges and repairs scattered conditional structures by identifying the cross-segment collocation relationship between conditional introductory words and result words, thereby improving the accuracy of the subsequent target recognition model during recognition.
[0186] Specifically, for each initial text segment, it is checked whether it meets a preset trigger condition. This preset trigger condition is configured to identify initial text segments that are semantically incomplete and require subsequent content completion, and may include at least one of the following two scenarios.
[0187] The first scenario: The initial text segment begins with a word from a pre-defined conditional vocabulary and its meaning is incomplete. The pre-configured conditional vocabulary can be a set of common conditional words, such as "if," "unless," "once," "suppose," "if," "as long as," etc. When the initial text segment begins with these conditional words, it usually means that the segment is the beginning of a conditional clause, and there must be a conclusive statement to be expressed afterward. However, simply starting with a conditional word is not enough to determine semantic incompleteness; it is also necessary to consider the auxiliary judgment of semantic completeness.
[0188] The second scenario: The initial text segment ends with a comma and its meaning is incomplete. In written Chinese, commas usually indicate a pause in tone and an incomplete continuation of meaning. Sentences ending with a comma are often grammatically subordinate clauses or adverbial phrases, indicating that a main clause must follow to complete their meaning. The determination of semantic incompleteness here can refer to the methods described above, using syntactic structure analysis to confirm whether the segment is a subordinate clause lacking a corresponding main clause.
[0189] Specifically, when an initial text segment is determined to meet any of the aforementioned preset trigger conditions, it is further checked whether the subsequent initial text segment contains words from a preset result vocabulary. The preset result vocabulary can be pre-configured as a set of common result-guiding words, such as "then," "will," "so," "that is," "then," "otherwise," "should," and "must." These words semantically function to connect with preceding conditions and introduce consequences, obligations, or judgments, serving as key identifiers for identifying the main result clause. If the subsequent initial text segment contains these result words, it indicates that the subsequent segment and the current segment form a logical condition-result pairing. In this case, the current initial text segment and the subsequent initial text segment are merged into a single semantic unit.
[0190] For example, the initial text segments "If one party violates the confidentiality obligations under this agreement" and "then shall bear the corresponding liability for breach of contract" obtained through preliminary sentence segmentation are detected. The first segment begins with the conditional introductory word "if" and is semantically incomplete, while the second segment begins with the result word "then". The merging condition is met. After merging the two, the semantic unit "If one party violates the confidentiality obligations under this agreement, then shall bear the corresponding liability for breach of contract" is obtained, which completely restores the conditional logical loop of the original text.
[0191] By merging the above-mentioned conditional integrity rules, the semantic logic breakage caused by the fragmentation of conditional sentences due to punctuation breaks is effectively prevented. This ensures that the final semantic unit is self-contained and complete in terms of conditional reasoning chain, providing a reliable basic semantic unit for the subsequent target recognition model to accurately understand the logical relationship between clauses and refine the target content, thereby improving the accuracy of subsequent recognition.
[0192] In the above implementation process, another implementation method for merging each initial text segment according to the preset semantic integrity rules to obtain multiple semantic units may include: for each initial text segment, detecting whether the initial text segment contains words from the preset enumeration guide dictionary; if the initial text segment contains words from the preset enumeration guide dictionary, then the initial text segment is determined as the enumeration start segment; starting from the enumeration start segment, sequentially detecting whether the next initial text segment meets the preset enumeration end condition; merging the enumeration start segment and all the next initial text segments until the preset enumeration end condition is met to obtain a semantic unit.
[0193] In formal texts such as contract terms, compliance review reports, and regulatory filings, enumeration structures are a very common way of organizing information, such as "Party B shall submit the following materials: a copy of its business license, proof of identity of its legal representative, and a power of attorney." Such enumeration structures are usually introduced by enumeration prompts, followed by several parallel enumeration items, which may be separated by commas, semicolons, or line breaks. However, in the initial sentence segmentation stage based on punctuation, the enumeration prompts and the subsequent enumeration items are easily segmented into multiple independent initial text segments, which may lead to the loss of the semantic relationship between the enumeration items and the prompts. This makes it difficult for subsequent target recognition models to accurately determine the nature and risk level of isolated enumeration items lacking context. To avoid the above problems, this embodiment provides an enumeration integrity rule. By using this rule to identify enumeration prompts and continuously merge subsequent related content, the enumeration structure fragmented by punctuation is re-aggregated into a self-contained semantic unit of complete enumeration information, thereby improving the recognition accuracy of subsequent models.
[0194] Specifically, for each initial text segment, it is detected whether the initial text segment contains words from a preset enumeration guide vocabulary. The preset enumeration guide vocabulary can be pre-configured to contain a set of common enumeration guide words, which may include, for example, "including but not limited to," "the following," "as follows," "below," "as follows," "has the following," "includes the following," etc. These words semantically clearly indicate that a series of parallel enumeration items will follow, and are key identifiers for identifying the starting position of the enumeration structure.
[0195] If an initial text segment is detected to contain a word from a preset enumeration guide dictionary, then that initial text segment is determined as the enumeration start segment. Starting from this enumeration start segment, each subsequent initial text segment is sequentially checked to see if it meets the preset enumeration end condition. If it does not meet the condition, the check continues; if it does meet the condition, the merging process stops. The preset enumeration end condition is used to define the termination boundary of the enumeration structure in the text and may include at least one of the following conditions.
[0196] Scenario 1: A word from a preset enumeration terminology library is detected. The preset enumeration terminology library can be pre-configured to contain a set of common enumeration terminators, such as "above," "etc.," "and so on," etc. These words typically function to conclude the enumeration content in the text. Once such an ending word is detected in a subsequent initial text segment, it indicates that the enumeration content has been fully presented, the merging operation should terminate, and the initial text segment containing the ending word should be included in the merging scope to maintain the semantic integrity of the enumeration.
[0197] Scenario 2: A new paragraph mark or document structure boundary is detected. When the subsequent initial text segment begins with a clear paragraph break (such as consecutive line breaks, blank lines, paragraph indentation changes, etc.), it indicates that there is a clear paragraph break in the original text at this point, and the enumerated content has ended. In addition, document format features (such as page breaks in Word documents, area jumps in PDF documents, etc.) can also be used as auxiliary criteria for judgment.
[0198] Scenario 3: A new title or topic start marker is detected. When the text format or content characteristics of the subsequent initial text segment indicate that it is a new chapter title, clause number, or independent topic sentence (e.g., a new paragraph starting in bold, numbered, or ordinal), it can be determined that the enumeration structure has terminated, and the current enumeration sequence should be closed.
[0199] After determining the starting segment of the enumeration and identifying the ending condition of the enumeration, the starting segment of the enumeration and all subsequent initial text segments from the starting segment of the enumeration until the preset ending condition of the enumeration are met are merged to form a semantic unit.
[0200] The above processing effectively prevents the enumerated items from being scattered into isolated fragments without context due to punctuation breaks. The resulting semantic units are self-contained with complete enumerated information and their associated context, providing a reliable processing foundation for the subsequent target recognition model to accurately understand the category and associated risks of each enumerated item, thereby improving recognition accuracy.
[0201] Based on the above implementation method, after merging each initial text segment to obtain multiple semantic units, the multiple semantic units can be filtered to obtain target semantic units, so that only high-value semantic segments are sent into the target recognition model in subsequent steps, thereby compressing the input scale and reducing the model call cost to the greatest extent while ensuring recognition accuracy.
[0202] In one embodiment, the process of semantic unit filtering may include: for each semantic unit, determining the expected token consumption and semantic point density of the semantic unit; and filtering at least one target semantic unit from multiple semantic units based on the expected token consumption and semantic point density of each semantic unit.
[0203] Specifically, the expected token consumption for each semantic unit is determined. The expected token consumption refers to the number of tokens expected to be consumed when the semantic unit is input into the target recognition model. This is calculated based on the character types and quantities contained in the semantic unit, combined with a preset token conversion rule. In one implementation, the number of Chinese characters and English words in the semantic unit can be counted. Chinese characters are converted according to a preset ratio, and English words are directly counted as a whole to obtain the basic token consumption for that semantic unit.
[0204] Furthermore, the semantic point density of each semantic unit is determined. Semantic point density is an indicator that measures the density of semantic features related to the target task recognition within a semantic unit, reflecting the amount of discriminative information carried by that semantic unit per unit token consumption. In one optional implementation, the determination of semantic point density can comprehensively consider the following factors: the number of entities contained in the semantic unit, the negation structure marker value of the semantic unit, and the proportion of stop words in the semantic unit. Based on the above factors and their respective preset weight coefficients, the semantic point density of the semantic unit can be calculated according to the preset point density calculation rules.
[0205] After obtaining the expected token consumption and semantic point density of each semantic unit, target semantic units can be selected from multiple semantic units based on these two indicators. For example, a greedy selection strategy based on cost-effectiveness ranking can be adopted. This involves calculating the cost-effectiveness score of each semantic unit, ranking them from highest to lowest, and selecting the top-ranked semantic units as target semantic units, provided that the token consumption threshold is not exceeded. This strategy ensures that semantic units with the highest information density and the greatest discriminative value are selected first within a limited token budget, achieving optimal selection under resource constraints.
[0206] In other alternative implementations, rule matching based on keyword databases or regular expressions can be used for filtering, directly identifying semantic units that match predefined target content as target semantic units; or a probability evaluation method based on a lightweight pre-trained classification model can be used to score the target content probability of each semantic unit, and identifying semantic units with scores higher than a preset threshold as target semantic units.
[0207] The above screening methods can be selected or combined according to the accuracy requirements and computing resource constraints of the actual application scenario. This embodiment does not impose specific limitations on this.
[0208] The above implementation method, by simultaneously considering the expected token consumption and semantic point density of semantic units for screening, can prioritize the retention of semantic units with high information density and high discriminative value under the constraint of limited token budget. This ensures the accuracy of target task recognition while compressing the input length, achieving the optimal balance between model calling cost and recognition effect.
[0209] In the above implementation process of determining the expected token consumption of a semantic unit, one implementation method may include: determining the number of first-type characters and the number of second-type characters in the semantic unit; wherein, the first-type characters are characters whose single-character token count conforms to a first preset rule, and the second-type characters are characters whose single-character token count conforms to a second preset rule; and determining the expected token consumption of the semantic unit based on the number of first-type characters, the number of second-type characters, and a preset token conversion rule.
[0210] Specifically, the number of first-class characters and the number of second-class characters in the semantic unit are determined. First-class characters are those whose token count conforms to a first preset rule, and second-class characters are those whose token count conforms to a second preset rule.
[0211] For example, the first type of characters can be Chinese characters, and the second type of characters can be English words. Chinese characters can be identified and counted by determining the Unicode range, for example, by determining whether the Unicode encoding of the character falls within the CJK Unified Ideographs range; the number of English words can be obtained by separating them with spaces.
[0212] It should be noted that the Chinese characters and English words mentioned above are only specific examples of the first and second categories of characters. The first category of characters may also include full-width characters such as Japanese kana and Korean Hangul, and the second category of characters may also include strings of numbers, punctuation marks, etc. This embodiment does not limit these.
[0213] After obtaining the number of characters in the first and second categories, the estimated token consumption of the semantic unit is calculated according to the preset token conversion rules. Specifically, the number of characters in the first category can be converted into a first token estimate according to a first conversion ratio, and the number of characters in the second category can be converted into a second token estimate according to a second conversion ratio. The sum of the first token estimate and the second token estimate is used as the base token consumption. Then, the base token consumption is multiplied by a preset calibration coefficient to obtain the estimated token consumption.
[0214] In one exemplary implementation, the conversion rule may include a representation using the following expression:
[0215] Ti = (CountChinese(ai) / 2.0 + CountEnglishWords(ai)) × K
[0216] Where Ti represents the expected token consumption of the i-th semantic unit, CountChinese(ai) represents the number of Chinese characters in the semantic unit, CountEnglishWords(ai) represents the number of English words in the semantic unit, and K is the preset calibration coefficient.
[0217] In the above expression, every two Chinese characters are converted into approximately one token, and every English word is converted into approximately one token. The sum of these two is then multiplied by a calibration coefficient K for correction. This empirical formula only requires counting the number of characters and words, resulting in extremely low computational complexity. It can complete token estimation for a single semantic unit within milliseconds, meeting the performance requirements of real-time preprocessing scenarios involving massive amounts of text.
[0218] In the above embodiments, the estimated token consumption of a semantic unit is directly estimated by classifying by character type in combination with a preset token conversion rule, and the estimation can be completed at the millisecond level without repeatedly calling the tokenizer of the target recognition model, thereby reducing the computational overhead and improving the processing speed in large-scale text preprocessing.
[0219] In the implementation process of determining the estimated token consumption of a semantic unit, in addition to the aforementioned rapid estimation method based on character type classification and empirical formulas, other solutions may also be adopted to determine the token consumption in some embodiments.
[0220] In an optional embodiment, the estimated token consumption may be directly estimated based on the model tokenizer. In this method, for each semantic unit, the tokenizer (Tokenizer) supporting the target recognition model is directly called to perform actual tokenization processing on the text content of the semantic unit, the number of tokens obtained after tokenization is counted, and the number of tokens is directly taken as the estimated token consumption of the semantic unit. Since the above method directly uses the native tokenization logic of the target recognition model, the obtained token consumption is completely consistent with the consumption during the actual processing of the model, thereby reducing the deviation introduced by the conversion rule or calibration coefficient in the token estimation step, and can provide a more accurate consumption basis for the subsequent semantic unit screening under the token budget constraint.
[0221] In another optional embodiment, a look-up table based method may also be used to rapidly estimate the estimated token consumption. In this method, a mapping table from characters or words to the number of tokens is pre-constructed. The mapping table may be obtained by counting the tokenization results of the target recognition model tokenizer on representative corpora, and the table records the average token consumption quantity corresponding to different characters, common words or subword units. For the semantic unit to be estimated, its text content is split by characters or tokenization units, the mapping table is queried item by item, and the number of tokens of each unit obtained from the query is accumulated to obtain the estimated token consumption of the semantic unit. As a result, the computational overhead of token estimation can be reduced without substantially losing the estimation accuracy, and the processing speed of large-scale text real-time preprocessing can be improved.
[0222] In practical applications, one of the methods may be selected or a plurality of methods may be used in combination according to factors such as accuracy requirements, computational resource constraints and real-time performance requirements, which is not specifically limited in this embodiment.
[0223] In this embodiment, the preset calibration coefficient is not a globally uniform fixed value, but a model-specific parameter corresponding to a specific target recognition model. Because different target recognition models use segmenters with varying word segmentation granularity and sub-word merging strategies, the actual number of tokens obtained after inputting the same text into different models often differs. Therefore, using the same calibration coefficient for all models may lead to systematic estimation bias. Based on this, this embodiment maintains its own preset calibration coefficient for each target recognition model, and initializes the preset calibration coefficients for each model to default values during system initialization. For example, it can be set to K=1.1. This value is an initial estimate based on statistical experience and can reflect, to some extent, the token segmentation ratio of common target recognition model segmenters for mixed Chinese and English text.
[0224] Furthermore, in scenarios requiring simultaneous service to multiple target recognition models, a mapping relationship between model identifiers and calibration coefficients can be established. This mapping can be stored as a key-value pair table maintained in a parameter repository or configuration center, with each record corresponding to a unique calibration coefficient for a target recognition model. When a received text processing request carries a target model identifier, the corresponding unique calibration coefficient for that model is automatically loaded from the mapping table based on the identifier, and this unique coefficient is used to calculate the estimated token consumption. If the target model identifier carried in the request does not exist in the mapping table, meaning a new, uncalibrated model has been matched, the default calibration coefficient (e.g., K=1.1) is used to complete the token estimation, ensuring the processing flow is not interrupted.
[0225] Building upon the above, since the word segmentation strategy of the target recognition model may drift due to factors such as model version updates and changes in the distribution of input text types, fixed calibration coefficients are difficult to maintain estimation accuracy throughout the long-term operation of the system. Therefore, this embodiment provides a calibration coefficient update strategy to adaptively maintain the accuracy of the coefficients during system operation, thereby improving estimation accuracy.
[0226] Based on the above implementation, one implementation of the update process for the preset calibration coefficient may include: every preset time period, obtaining the actual token consumption of the text to be recognized in the target recognition model within the time period, and the expected token consumption calculated according to the preset token conversion rule; if the difference between the actual token consumption and the expected token consumption is greater than a preset threshold, then the preset calibration coefficient is updated; if the difference is less than or equal to the preset threshold, then the current value of the preset calibration coefficient remains unchanged.
[0227] Specifically, at preset time intervals, the actual token consumption of the text to be recognized in the target recognition model within that time interval, and the expected token consumption calculated according to preset token conversion rules, are obtained. The actual token consumption can be directly obtained from the token usage statistics field in the target recognition model call results, while the expected token consumption is calculated based on the aforementioned number of characters in the first category, the number of characters in the second category, and the current preset calibration coefficient, according to the preset token conversion rules. The preset time interval can be flexibly set according to the actual business processing volume or time interval, such as per hour, per thousand texts processed, or per batch of recognition tasks completed.
[0228] After obtaining the two consumption figures, the difference between them is calculated and compared with a preset threshold. If the difference between the actual token consumption and the expected token consumption is greater than the preset threshold, it indicates a significant deviation between the current preset calibration coefficient and the actual word segmentation consumption characteristics, requiring an update to the preset calibration coefficient so that subsequent token estimation results approximate the true consumption level. If the difference is less than or equal to the preset threshold, it indicates that the current preset calibration coefficient still reflects the actual consumption well and no adjustment is needed. The current value of the preset calibration coefficient should remain unchanged to avoid coefficient oscillations and system instability caused by frequent updates.
[0229] Through the aforementioned periodic detection and condition updates, the preset calibration coefficients can continuously monitor the actual word segmentation features of the target recognition model. While ensuring the long-term stability of estimation accuracy, it avoids unnecessary frequent adjustments and provides a reliable and stable basis for subsequent semantic unit screening based on token budget constraints.
[0230] In the above process, if a significant deviation is detected between the current preset calibration coefficient and the actual word segmentation consumption features, the preset calibration coefficient is updated. In one embodiment, the update process of the preset calibration coefficient may include: constructing a calibration verification set; the calibration verification set contains multiple samples extracted from the processed text; for each sample in the calibration verification set, calculating the basic token consumption and actual token consumption for each sample; summing the basic token consumption and actual token consumption for all samples to obtain the total basic token consumption and the total actual token consumption; determining candidate calibration coefficients based on the total basic token consumption and the total actual token consumption; if the difference between the ratio of the total actual token consumption to the total basic token consumption and the current preset calibration coefficient is greater than a preset convergence threshold, then a smoothing update strategy is used to process the current preset calibration coefficient based on the ratio to obtain the updated preset calibration coefficient.
[0231] When updating the preset calibration coefficient, a calibration verification set is constructed. This calibration verification set contains multiple samples extracted from recently processed texts. The samples can be extracted using a stratified random sampling strategy, and representative texts are selected from service logs as calibration samples. To ensure the universality of the calibration results for different text types, the sampling shall cover core service scenarios, such as credit contracts, regulatory reports, legal documents, etc., and be representative in terms of character length, Chinese-English proportion, professional term density and the like. The number of samples in the calibration verification set can be set according to actual needs, for example, 50 to 100 samples can be selected to strike a balance between calculation overhead and calibration accuracy.
[0232] After the calibration verification set is constructed, for each sample in the calibration verification set, the basic token consumption amount and the actual token consumption amount of each sample are calculated respectively. In one implementation, the basic token consumption amount can be obtained by traversing each sample in the verification set and calculating according to the aforementioned empirical formula (for example, the number of Chinese characters divided by 2.0 plus the number of English words). The actual token consumption amount can be obtained by inputting the texts of the verification set into the official tokenizer supporting the target model in batches.
[0233] On this basis, the basic token consumption amounts of all samples are summed to obtain the total basic token consumption amount; the actual token consumption amounts of all samples are summed to obtain the total actual token consumption amount. The ratio of the total actual token consumption amount to the total basic token consumption amount is calculated, and this ratio is determined as the candidate calibration coefficient. Here, the candidate calibration coefficient can reflect the ideal calibration ratio presented by the current verification set without considering historical coefficients.
[0234] After the candidate calibration coefficient is obtained, a difference value between the candidate calibration coefficient and the current preset calibration coefficient is determined. If the difference value is greater than the preset convergence threshold, it indicates that the system has not converged and the current coefficient needs to be adjusted; at this time, a smoothing update strategy is adopted to process the current preset calibration coefficient according to the candidate calibration coefficient, so as to obtain the updated preset calibration coefficient. In one implementation, the smoothing update strategy may adopt an exponential moving average method, that is, weighted fusion is performed on the current coefficient and the candidate coefficient according to a preset smoothing attenuation factor (for example, 0.7 to 0.8), so as to prevent coefficient oscillation caused by single sampling fluctuation. If the difference value is less than or equal to the preset convergence threshold, it is determined that the system has converged, the configuration is not updated this time, and the current preset calibration coefficient remains unchanged. The updated preset calibration coefficient can be persisted to the parameter repository, and immediately issued to all preprocessing nodes to take effect, thereby realizing the consistency and accuracy of subsequent token estimation.
[0235] In the above embodiment, by constructing a calibration verification set, a candidate calibration coefficient is determined using the ratio of the actual total token consumption to the basic total token consumption, and a smooth update strategy is adopted to correct the current coefficient when the difference exceeds the convergence threshold, so that the preset calibration coefficient can be adaptively calibrated along with the real word segmentation result, while suppressing coefficient oscillation caused by single sampling fluctuation, continuously improving the estimation accuracy of the estimated token consumption, thereby achieving the improvement of the consistency and accuracy of subsequent token estimation.
[0236] In the above implementation process of calculating the estimated token consumption of a semantic unit, this embodiment also synchronously calculates the semantic key point density of the semantic unit, so as to quantitatively evaluate each semantic unit from the dimension of information value. An optional embodiment may include: determining the number of entities, the negative structure flag value, and the stop word proportion in the semantic unit; wherein if the semantic unit is marked as containing a negative structure in the semantic segmentation stage, the negative structure flag value is a first preset value, otherwise it is a second preset value; the stop word proportion is the ratio of the number of stop words to the total number of words in the semantic unit; determining the semantic key point density of the semantic unit based on the number of entities, the negative structure flag value, the stop word proportion, their respective corresponding preset weight coefficients, and a preset key point density calculation rule.
[0237] In this embodiment, the number of entities refers to the number of named entities contained in the semantic unit. Named entities may include types such as person names, place names, organization names, amounts, dates, etc., and their extraction can be implemented through a lightweight regular matching library or a pre-trained small named entity recognition model. For example, amounts can be recognized by matching numbers followed by units such as "Yuan" and "Ten Thousand Yuan" through regular expressions, dates can be recognized by regularly matching common date formats, and person names, place names, organization names, etc. can be counted after being labeled by the named entity recognition model. A larger number of entities usually indicates that the semantic unit contains more substantial information.
[0238] The negative structure flag value is derived from the marking result of whether the semantic unit contains a negative structure in the semantic segmentation stage. If the semantic unit is marked as containing a negative structure during segmentation or merging processing, the negative structure flag value takes a first preset value (such as 1), otherwise it takes a second preset value (such as 0). Since the presence of a negative structure may fundamentally change the semantic tendency of a semantic unit and has extremely high discriminative value for target task recognition, a higher weight should be assigned in the calculation of semantic key point density to prevent semantic misjudgment caused by ignoring the negative structure.
[0239] The stop word proportion refers to the ratio of the number of stop words to the total number of words in the semantic unit. Stop words may include common function words without actual meanings such as "de", "le", "he", "yu", "ji", etc. A higher stop word proportion indicates that there are more redundant components in the semantic unit and the effective information density is lower, so it should be treated as a subtraction item in the calculation of semantic key point density.
[0240] Based on the aforementioned entity count, negation structure marker value, and stop word ratio, combined with their respective preset weight coefficients and preset key point density calculation rules, the semantic key point density of the semantic unit can be determined. In one specific implementation, the semantic key point density can be calculated using the following formula:
[0241]
[0242] Wherein, Scori represents the semantic point density of the i-th semantic unit; EntityCount(ai) represents the number of named entities in the semantic unit; NegationFlag(ai) represents the negation structure flag value; StopWordRatio(ai) represents the stop word ratio; W1, W2, and P are preset weight coefficients corresponding to the number of entities, the negation structure flag value, and the stop word ratio, respectively. For example, W1 can be set to 2.0 to reflect the positive contribution of entity information to target recognition; W2 can be set to 3.0 to emphasize the potential impact of negation structure on semantic inversion with a higher weight; and P can be set to 1.0 as a penalty coefficient for the stop word ratio. It should be understood that the specific values of each weight coefficient can be adjusted according to the actual application scenario and the type of target content, and this embodiment does not impose specific limitations on this.
[0243] In the above implementation, by comprehensively calculating the semantic point density by weighting the number of entities, the negative structure marker value, and the proportion of stop words, the target recognition value of each semantic unit can be more accurately quantified, providing a reliable basis for subsequent screening. Thus, under a limited token budget, high-value semantic units are prioritized for retention, reducing the risk of missed detection and improving recognition efficiency.
[0244] In one optional implementation, for certain business scenarios, the process of determining the semantic point density of semantic units can be adjusted differently based on the specific business scenario.
[0245] For example, in a Data Loss Prevention (DLP) scenario, the initial semantic point density calculated solely based on the number of entities, negative structures, and the proportion of stop words, while reflecting the information density of semantic units to some extent, cannot fully capture certain contextual features highly relevant to the risk of data leakage. Therefore, this embodiment, based on the initial semantic point density, further introduces a contextual word detection and adjustment mechanism specifically for DLP scenarios to enhance the semantic point density's ability to distinguish sensitive content.
[0246] Specifically, the initial semantic point density of the semantic unit is obtained. This initial semantic point density can be calculated based on the number of entities, the negation structure marker value, and the proportion of stop words in the aforementioned embodiments, according to the corresponding preset weight coefficients and preset point density calculation rules, which will not be elaborated here. For ease of distinction, this initial semantic point density can be denoted as Score_base in this scenario.
[0247] After obtaining the initial semantic point density, it is detected whether the semantic unit contains a first type of preset contextual words and / or a second type of preset contextual words. The first and second types of preset contextual words can be pre-configured according to specific business scenarios. For example, in a DLP scenario, the first type of preset contextual words can be configured as high-risk contextual words, such as "leakage," "violation," "unauthorized," "unauthorized copying," "external distribution," and "exceeding authority." The presence of these words usually indicates that the behavior or fact described by the semantic unit has a high risk of data leakage. The second type of preset contextual words can be configured as compliance exemption contextual words, such as "approved," "whitelisted," "after de-identification," "authorized," and "compliant." The presence of these words usually indicates that the relevant behavior has undergone compliance procedures and its actual risk is low, and it should not be over-labeled.
[0248] Based on this, the final semantic point density of the semantic unit in the DLP scenario is calculated according to the detection results, the initial semantic point density, and the preset semantic point density adjustment rules.
[0249] In one exemplary implementation, the adjustment rule can be expressed by the following formula: Score_DLP = Score_base + (RiskContextFlag × W3) + (ExemptionFlag × W4).
[0250] Wherein, Score_DLP is the semantic point density after DLP scenario adjustment; Score_base is the initial semantic point density; RiskContextFlag is the hit flag for the first type of preset context words (high-risk context words), which is 1 when hit and 0 when not hit; ExemptionFlag is the hit flag for the second type of preset context words (compliance exemption context words), which is 1 when hit and 0 when not hit; W3 is the first preset weight value, which corresponds to the positive correction magnitude of the semantic point density when high-risk context words are hit, and can be set to 2.5 for example; W4 is the second preset weight value, which corresponds to the negative correction magnitude of the semantic point density when compliance exemption context words are hit, and can be set to -1.5 for example.
[0251] As can be seen from the above formula, when a semantic unit contains high-risk contextual words, its semantic point density receives a positive bonus based on its initial value, making it more likely to be prioritized in subsequent screening and ranking, ensuring that high-risk content is not overlooked. Conversely, when a semantic unit contains compliance exemption contextual words, its semantic point density is reduced from its initial value, thus placing it appropriately lower in the screening and ranking process, avoiding the duplication of already compliant content as high-risk content, and reducing false positives. When the same semantic unit contains both types of contextual words simultaneously, the positive bonus and negative penalty take effect simultaneously, and the final score depends on the combined effect of the two, thus more accurately reflecting the true risk level of the semantic unit.
[0252] Through the above implementation methods, in specific business scenarios, such as DLP scenarios, risk context priors are introduced on the basis of the initial semantic point density. Positive points are added to high-risk context words and negative points are deducted for compliance exemption context words. This makes the semantic point density more in line with the actual business risk judgment needs, thereby allocating the limited token budget more accurately to semantic units with real risk judgment value in subsequent screening and improving the recognition accuracy.
[0253] After calculating the expected token consumption and semantic point density of the expected semantic units, at least one target semantic unit is selected from them based on the above two factors of each semantic unit.
[0254] In one implementation, the screening process may include: for each semantic unit, calculating the cost-effectiveness score of the semantic unit based on its expected token consumption and semantic point density; sorting multiple semantic units in descending order of cost-effectiveness score to obtain a sorted semantic unit sequence; starting from the first semantic unit in the semantic unit sequence, accumulating the expected token consumption of each semantic unit sequentially to obtain a cumulative token consumption; if the cumulative token consumption is less than a preset token consumption threshold, then continuing to accumulate the expected token consumption of the next semantic unit; if the cumulative token consumption is greater than or equal to the preset token consumption threshold, then stopping the accumulation, and determining all semantic units before the semantic unit involved in the current accumulation as target semantic units.
[0255] In this embodiment, the process of selecting at least one target semantic unit from multiple semantic units can adopt a greedy selection strategy based on cost-effectiveness ranking. Under the constraint of a preset token budget, the semantic unit with the highest semantic discriminative value brought by the unit token consumption is selected first, i.e., the target semantic unit.
[0256] Specifically, a target token budget is set, which is a preset token consumption threshold T_budget. This threshold can be set based on the maximum number of input tokens allowed in a single call of the target recognition model, the cost budget, or the system resource status; for example, it can be set to 500 tokens. The size of this threshold determines the total scale of the final input content and is a resource constraint that must be strictly adhered to during the screening process.
[0257] For each semantic unit, a cost-effectiveness score (Ratioi) is calculated based on its projected token consumption (Ti) and semantic point density (Scorei) determined in the previous steps. The cost-effectiveness score characterizes the semantic discriminative value that the semantic unit can provide per unit token consumption. In one specific implementation, the cost-effectiveness score can be calculated using the following formula:
[0258] Ratioi = Scorei / (Ti + 0.1)
[0259] Where Ti is the expected token consumption of the i-th semantic unit, and Scori is the semantic point density of the i-th semantic unit. The constant 0.1 is added to the denominator to prevent division by zero when Ti is zero. The value of this constant is extremely small and will not have a substantial impact on the ranking result of the cost-effectiveness score.
[0260] After obtaining the cost-effectiveness score for each semantic unit, the semantic units are sorted in descending order of their cost-effectiveness scores, resulting in a sorted sequence of semantic units. This sorting operation places the semantic units with the highest discriminative value density at the beginning of the sequence, ensuring that these high-value units are prioritized in subsequent selection processes.
[0261] Based on this, the selected semantic unit set is initialized to an empty set, and the current cumulative token consumption is set to zero. Starting from the first semantic unit in the sorted semantic unit sequence, the following judgments are performed sequentially: if the sum of the current cumulative token consumption and the expected token consumption of the current semantic unit is less than or equal to the preset token consumption threshold T_budget, then the semantic unit is added to the selected semantic unit set, and the expected token consumption of the semantic unit is added to the current cumulative token consumption, and then the next semantic unit in the sequence is processed.
[0262] If the sum of the current cumulative token consumption and the expected token consumption of the current semantic unit exceeds the preset token consumption threshold T_budget, then accumulation stops, all semantic units in the selected semantic unit set are identified as target semantic units, and the filtering process ends. The semantic unit currently being processed and all subsequent semantic units are no longer selected and are directly discarded.
[0263] Through the above process, under the premise of not exceeding the preset token consumption threshold, several semantic units with the highest cost-effectiveness scores were selected first, so that the limited token budget was allocated to the semantic content with the most discriminative value.
[0264] In an optional implementation, when the sum of the current cumulative token consumption and the expected token consumption of the current semantic unit exceeds a preset threshold, the semantic unit can be skipped, and subsequent semantic units in the sequence with smaller expected token consumption can be tried to make fuller use of the remaining token budget, until the entire sequence has been traversed or no more semantic units can be added. This method and the above-described stopping accumulation method can be selected according to the different needs of budget utilization and computational efficiency in the actual application scenario, and this embodiment does not specifically limit this.
[0265] In other implementations, the strategy for filtering target semantic units from multiple semantic units may also include the following other schemes.
[0266] For example, an optimal allocation method based on dynamic programming can be used. In this method, the semantic unit selection problem is modeled as a combinatorial optimization problem that maximizes the sum of semantic point densities under a preset token budget constraint. The globally optimal subset is solved by dynamic programming, thereby obtaining the theoretically optimal selection result within a finite budget.
[0267] For example, an integer linear programming-based selection method can be used. In this method, the selection criteria are formalized into an integer linear programming model, with whether a semantic unit is selected as the decision variable, and the optimal selection scheme is solved under constraints such as token budget and semantic point density.
[0268] In addition, rule-based filtering based on a keyword database is used to match semantic units using a pre-defined sensitive keyword database or regular expressions. Semantic units that match predefined sensitive patterns are directly identified as target semantic units, enabling rapid filtering.
[0269] Building upon the above implementation, a probability evaluation and screening method based on a lightweight pre-trained classification model is also employed. This method utilizes a pre-trained lightweight classification model to evaluate the probability that each semantic unit contains sensitive content, and identifies semantic units with probabilities higher than a preset threshold as target semantic units.
[0270] It should be understood that the above screening strategies can be used individually or in combination according to different requirements for accuracy, computational efficiency and interpretability in actual application scenarios, and this embodiment does not make specific limitations in this regard.
[0271] Given the target semantic unit obtained based on the above implementation method, input content that meets the input requirements of the target recognition model is generated based on the target semantic unit.
[0272] In one implementation, the original semantic position of each target semantic unit in the text to be recognized is obtained; the target semantic units are sorted according to the order of their original semantic position representations to obtain a sorted sequence of target semantic units; the target semantic units in the sequence are connected sequentially using a preset connector to obtain a connected text; and the connected text is encapsulated using a preset prompt template to obtain the input content.
[0273] Specifically, the original semantic position of each target semantic unit in the text to be recognized is obtained. This original semantic position can be represented by index information such as the starting character offset, sentence number, or paragraph position of the target semantic unit in the original text to be recognized. It is recorded during the semantic segmentation stage and saved along with the semantic units. According to the order represented by the original semantic position, the target semantic units are sorted to obtain a sorted sequence of target semantic units. Since the target semantic units may deviate from the original text order during the screening stage due to sorting by cost-effectiveness score, the sorting and restoration step here ensures that the final generated input content logically maintains the original text's chronological or argumentative order, avoiding the impact of disordered order on the target recognition model's understanding of the contextual logic.
[0274] After obtaining the sorted sequence of target semantic units, pre-defined connectors are used to sequentially connect the target semantic units in the sequence, resulting in connected text. Pre-defined connectors are used to insert clear delimiters between adjacent target semantic units, enabling the model to distinguish the boundaries between different semantic units while maintaining the overall readability of the text. In one implementation, the pre-defined connector can be set to a special delimiter, such as "|||", a symbol that rarely appears in natural text and can effectively separate elements without being confused with the main text. The connected text is the concatenated result containing the content of all selected target semantic units.
[0275] Building upon this, the joining text is encapsulated using a preset prompt template to obtain the input content. The preset prompt template provides task instructions and contextual guidance to the object recognition model, enabling the model to clearly understand the type of task it needs to perform. In one implementation, the preset prompt template can be set to: "The following are key fact fragments of the document; please determine whether there is any violation based on this information: {joined_text}", where {joined_text} is a placeholder for the aforementioned joining text. By embedding the joining text into this template, the final input content string is generated. This input content can be directly used as API input for the object recognition model (e.g., a large language model) and sent to the model for object task recognition.
[0276] Through the above-mentioned sorting restoration, delimiter connection, and prompt template encapsulation processes, the originally discrete target semantic units, which may be out of order after filtering, are reorganized into a logically coherent, formatted input content that carries task instructions. This ensures the semantic integrity and logical order of the information received by the model, and guides the model to focus on the target task recognition task through prompt templates, thereby improving the relevance and accuracy of the model output.
[0277] Figure 4 A schematic diagram of the structure of the text processing device provided in this application is shown below. Figure 4 As shown, the text processing device 40 provided in this embodiment includes:
[0278] The semantic unit acquisition module 401 is used to acquire the text to be recognized, perform semantic segmentation on the text to be recognized, and obtain multiple semantic units; wherein, each semantic unit expresses an independent and complete semantic meaning;
[0279] The semantic unit filtering module 402 is used to filter out at least one target semantic unit from multiple semantic units;
[0280] The input content generation module 403 is used to generate target input content based on the target semantic unit; wherein, the target input content is content that meets the input requirements of the target recognition model and is intended for target task recognition by the target recognition model.
[0281] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0282] For each semantic unit, determine the expected token consumption and semantic point density of the semantic unit;
[0283] Based on the expected token consumption and semantic point density of each semantic unit, at least one target semantic unit is selected from multiple semantic units.
[0284] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0285] Determine the number of first-class characters and second-class characters in the semantic unit; wherein, the first-class characters are characters whose token count as a single character conforms to the first preset rule, and the second-class characters are characters whose token count as a single character conforms to the second preset rule;
[0286] Based on the number of characters in the first category, the number of characters in the second category, and the preset token conversion rules, the expected token consumption of the semantic unit is determined.
[0287] In one possible implementation, the token conversion rule includes a preset calibration coefficient; the semantic unit acquisition module 401 is further used for:
[0288] The preset calibration coefficients are initialized to the default values. The preset calibration coefficients correspond to the target recognition model. Different target recognition models have their own preset calibration coefficients, and the default values are the default values corresponding to the target recognition model.
[0289] Every preset time period, obtain the actual token consumption of the text to be recognized in the target recognition model within the time period, as well as the expected token consumption calculated according to the preset token conversion rule;
[0290] If the difference between the actual token consumption and the expected token consumption is greater than the preset threshold, the preset calibration coefficient will be updated.
[0291] If the difference is less than or equal to the preset threshold, the current value of the preset calibration coefficient remains unchanged.
[0292] In one possible implementation, the semantic unit acquisition module 401 is further configured to:
[0293] Construct a calibration validation set; the calibration validation set contains multiple samples extracted from the processed text;
[0294] For each sample in the calibration verification set, calculate the base token consumption and the actual token consumption for each sample;
[0295] Sum the base token consumption and the actual token consumption for all samples to obtain the total base token consumption and the total actual token consumption.
[0296] Candidate calibration coefficients are determined based on the base token consumption and the total actual token consumption.
[0297] If the difference between the ratio of the actual total token consumption to the total basic token consumption and the current preset calibration coefficient is greater than the preset convergence threshold, then the current preset calibration coefficient is processed according to the ratio using a smooth update strategy to obtain the updated preset calibration coefficient.
[0298] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0299] Determine the number of entities, the negation structure marker value, and the proportion of stop words in the semantic unit;
[0300] Wherein, if a semantic unit is marked as containing a negation structure during the semantic segmentation stage, the negation structure marking value is the first preset value; otherwise, it is the second preset value; the stop word ratio is the ratio of the number of stop words in the semantic unit to the total number of words;
[0301] The semantic point density of a semantic unit is determined based on the number of entities, the negation structure marker value, the proportion of stop words, their respective preset weight coefficients, and the preset point density calculation rules.
[0302] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0303] Obtain the initial semantic point density of the semantic unit; the initial semantic point density is calculated based on the number of entities, the negation structure marker value, and the proportion of stop words;
[0304] Detect whether the semantic unit contains a first type of preset context word and / or a second type of preset context word;
[0305] Based on the detection results, the initial semantic point density, and the preset semantic point density adjustment rules, the semantic point density of the semantic unit is calculated.
[0306] The preset semantic point density adjustment rules include: if a semantic unit contains a first type of preset context word, then the first preset weight value is increased based on the initial semantic point density; if a semantic unit contains a second type of preset context word, then the second preset weight value is decreased based on the initial semantic point density; the first preset weight value is positive and the second preset weight value is negative.
[0307] In one possible implementation, the semantic unit filtering module 402 is specifically used for:
[0308] For each semantic unit, the cost-effectiveness score of the semantic unit is calculated based on the expected token consumption and semantic point density of the semantic unit.
[0309] The semantic units are sorted in descending order of cost-effectiveness score to obtain a sorted sequence of semantic units.
[0310] Starting from the first semantic unit in the semantic unit sequence, the expected token consumption of each semantic unit is accumulated sequentially to obtain the cumulative token consumption;
[0311] If the cumulative token consumption is less than the preset token consumption threshold, then the expected token consumption of the next semantic unit will continue to be accumulated.
[0312] If the cumulative token consumption is greater than or equal to the preset token consumption threshold, the accumulation will stop, and all semantic units preceding the semantic unit involved in the current accumulation will be identified as the target semantic unit.
[0313] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0314] Obtain the original content to be identified; the original content includes at least one of text content, image content, and table content;
[0315] Based on the text extraction strategy corresponding to the content type of the original content, the original content is processed to extract text and obtain the text to be identified.
[0316] In one possible implementation, the original content includes table content; the semantic unit acquisition module 401 is specifically used for:
[0317] Identify the header and data areas within a table.
[0318] Based on the field identifiers in the header area, each data record in the data area is converted into a text statement in key-value pair form to obtain the text to be recognized;
[0319] Specifically, when generating text to be recognized, a priority retention flag is set for the text content corresponding to the header area; the priority retention flag is used to indicate that when filtering target semantic units from semantic units to generate input content, the semantic units corresponding to the header area are retained first.
[0320] In one possible implementation, the original content includes text content; the semantic unit acquisition module 401 is specifically used for:
[0321] Obtain multiple text blocks from the text content and calculate the content features of each text block; the content features include at least one of hash features and similarity features;
[0322] Text blocks whose content features meet the preset repetition conditions are identified as redundant text blocks, and redundant text blocks are filtered out to obtain the text to be recognized after removing redundancy.
[0323] In one possible implementation, the original content includes image content; the semantic unit acquisition module 401 is specifically used for:
[0324] Optical character recognition is performed on the image content to obtain the initial recognized text and the page position information corresponding to the initial recognized text;
[0325] Determine the text segmentation boundaries based on the page layout information and / or the format control characters contained in the initially identified text;
[0326] The initial text to be recognized is segmented at the text segmentation boundary to obtain the text to be recognized.
[0327] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0328] Based on punctuation marks, the text to be recognized is segmented into multiple initial text segments;
[0329] According to the preset semantic integrity rules, the initial text segments are merged to obtain multiple semantic units.
[0330] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0331] For each initial text segment, check whether the initial text segment contains a negative word from a preset negative word library;
[0332] If the initial text segment contains a negative word, then determine whether the initial text segment and the subsequent initial text segment satisfy the conditions of subject agreement or subject omission;
[0333] If the conditions are met, the initial text segment is merged with the next initial text segment to obtain a semantic unit.
[0334] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0335] For each initial text segment, check whether the initial text segment meets the preset trigger conditions; wherein, the preset trigger conditions include: the initial text segment begins with a word in the preset condition-guided dictionary and the semantics are incomplete, or the initial text segment ends with a comma and the semantics are incomplete;
[0336] If the initial text segment meets the preset trigger condition, then check whether the next initial text segment contains a word from the preset result dictionary.
[0337] If included, the initial text segment is merged with the next initial text segment to obtain a semantic unit.
[0338] In one possible implementation, the semantic unit acquisition module 401 is specifically used for:
[0339] For each initial text segment, check whether the initial text segment contains words from the pre-defined guided vocabulary;
[0340] If the initial text segment contains words from the preset enumerated guide vocabulary, then the initial text segment is determined as the enumeration starting segment;
[0341] Starting from the first segment of the enumeration, check in turn whether the next initial text segment meets the preset enumeration end condition;
[0342] The first segment of the enumeration and all subsequent initial text segments up to and including the one that meets the preset enumeration end condition are merged to obtain a semantic unit.
[0343] In one possible implementation, the input content generation module 403 is specifically used for:
[0344] Obtain the original semantic position of each target semantic unit in the text to be recognized;
[0345] The target semantic units are sorted according to the order of their original semantic position representations to obtain a sorted sequence of target semantic units.
[0346] Using preset connectors, the target semantic units in the target semantic unit sequence are connected sequentially to obtain the connected text;
[0347] The input content is obtained by encapsulating the link text using a preset prompt template.
[0348] The text processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0349] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0350] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.
[0351] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0352] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0353] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0354] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0355] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0356] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0357] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0358] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0359] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0360] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0361] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0362] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0363] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0364] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A text processing method, characterized in that, include: The text to be identified is obtained, and the text to be identified is semantically segmented to obtain multiple semantic units; wherein each semantic unit expresses an independent and complete semantic meaning; At least one target semantic unit is selected from the plurality of semantic units; the target semantic unit is the selected semantic unit used as input content for constructing the target recognition model; Based on the target semantic unit, input content that meets the input requirements of the target recognition model is generated; wherein, the input content is configured to trigger the target recognition model to perform target task recognition after being input into the target recognition model.
2. The method according to claim 1, characterized in that, The step of selecting at least one target semantic unit from the plurality of semantic units includes: For each semantic unit, determine the expected token consumption and semantic point density of the semantic unit; Based on the expected token consumption and semantic point density of each semantic unit, at least one target semantic unit is selected from the plurality of semantic units.
3. The method according to claim 2, characterized in that, Determining the expected token consumption of the semantic unit includes: Determine the number of first-type characters and the number of second-type characters in the semantic unit; wherein, the first-type characters are characters whose token count as a single character conforms to a first preset rule, and the second-type characters are characters whose token count as a single character conforms to a second preset rule; The expected token consumption of the semantic unit is determined based on the number of characters in the first type, the number of characters in the second type, and the preset token conversion rules.
4. The method according to claim 3, characterized in that, The token conversion rules include a preset calibration coefficient; The method further includes: The preset calibration coefficients are initialized to default values; wherein, the preset calibration coefficients correspond to the target recognition model, different target recognition models have their own preset calibration coefficients, and the default values are the default values corresponding to the target recognition model; Every preset time period, the actual token consumption of the text to be identified in the target recognition model within the time period is obtained, as well as the expected token consumption calculated according to the preset token conversion rule; If the difference between the actual token consumption and the expected token consumption is greater than a preset threshold, the preset calibration coefficient is updated. If the difference is less than or equal to the preset threshold, the current value of the preset calibration coefficient remains unchanged.
5. The method according to claim 4, characterized in that, The step of updating the preset calibration coefficient includes: Construct a calibration verification set; the calibration verification set contains multiple samples extracted from the processed text; For each sample in the calibration verification set, calculate the base token consumption and the actual token consumption for each sample; Sum the base token consumption and the actual token consumption for all the samples to obtain the total base token consumption and the total actual token consumption. Candidate calibration coefficients are determined based on the base token consumption and the total actual token consumption. If the difference between the ratio of the actual total token consumption to the basic total token consumption and the current preset calibration coefficient is greater than the preset convergence threshold, then the current preset calibration coefficient is processed according to the ratio using a smooth update strategy to obtain the updated preset calibration coefficient.
6. The method according to claim 2, characterized in that, Determining the semantic point density of the semantic unit includes: Determine the number of entities, the negation structure marker value, and the proportion of stop words in the semantic unit; Wherein, if the semantic unit is marked as containing a negation structure during the semantic segmentation stage, the negation structure marking value is a first preset value; otherwise, it is a second preset value; the stop word ratio is the ratio of the number of stop words in the semantic unit to the total number of words. The semantic point density of the semantic unit is determined based on the number of entities, the negation structure marker value, the proportion of stop words, their respective preset weight coefficients, and the preset point density calculation rules.
7. The method according to claim 2, characterized in that, Determining the semantic point density of the semantic unit includes: Obtain the initial semantic point density of the semantic unit; the initial semantic point density is calculated based on the number of entities, the negation structure marker value, and the proportion of stop words in the semantic unit; Detect whether the semantic unit contains a first type of preset context word and / or a second type of preset context word; Based on the detection results, the initial semantic point density, and the preset semantic point density adjustment rules, the semantic point density of the semantic unit is calculated. The preset semantic point density adjustment rule includes: if the semantic unit contains the first type of preset context words, then a first preset weight value is added to the initial semantic point density; if the semantic unit contains the second type of preset context words, then a second preset weight value is reduced to the initial semantic point density; the first preset weight value is positive and the second preset weight value is negative.
8. The method according to claim 2, characterized in that, Based on the expected token consumption and semantic point density of each semantic unit, at least one target semantic unit is selected from the plurality of semantic units, including: For each semantic unit, a cost-effectiveness score is calculated based on the expected token consumption and the semantic point density of the semantic unit. The semantic units are sorted in descending order of their cost-effectiveness scores to obtain a sorted sequence of semantic units. Starting from the first semantic unit in the semantic unit sequence, the expected token consumption of each semantic unit is accumulated sequentially to obtain the cumulative token consumption; If the cumulative token consumption is less than the preset token consumption threshold, then the expected token consumption of the next semantic unit continues to be accumulated; If the cumulative token consumption is greater than or equal to the preset token consumption threshold, then the accumulation stops, and all semantic units preceding the semantic unit involved in the current accumulation are determined as the target semantic unit.
9. The method according to claim 1, characterized in that, The process of obtaining the text to be recognized includes: Obtain the original content to be identified; the original content includes at least one of text content, image content, and table content; Based on the text extraction strategy corresponding to the content type of the original content, the original content is processed to extract text, thereby obtaining the text to be identified.
10. The method according to claim 9, characterized in that, The original content includes the table content; The text extraction strategy based on the content type of the original content is used to perform text extraction processing on the original content to obtain the text to be identified, including: Identify the header area and data area in the table content; Based on the field identifiers in the header area, each data record in the data area is converted into a text statement in key-value pair form to obtain the text to be identified; Specifically, when generating the text to be recognized, a priority retention flag is set for the text content corresponding to the header area; the priority retention flag is used to indicate that when filtering target semantic units from semantic units to generate the input content, the semantic units corresponding to the header area are retained first.
11. The method according to claim 9, characterized in that, The original content includes text content; The text extraction strategy based on the content type of the original content is used to perform text extraction processing on the original content to obtain the text to be identified, including: Multiple text blocks are obtained from the text content, and the content features of each text block are calculated; the content features include at least one of hash features and similarity features. Text blocks whose content features meet the preset repetition conditions are identified as redundant text blocks, and the redundant text blocks are filtered out to obtain the text to be identified after removing redundancy.
12. The method according to claim 9, characterized in that, The original content includes image content; The text extraction strategy based on the content type of the original content is used to perform text extraction processing on the original content to obtain the text to be identified, including: Optical character recognition is performed on the image content to obtain initial recognized text and the page position information corresponding to the initial recognized text; The text segmentation boundaries are determined based on the layout position information and / or the format control characters contained in the initial identified text; The initial recognition text is segmented at the text segmentation boundary to obtain the text to be recognized.
13. The method according to claim 1, characterized in that, The semantic segmentation of the text to be identified yields multiple semantic units, including: Based on punctuation marks, the text to be identified is segmented into multiple initial text segments; According to the preset semantic integrity rules, the initial text segments are merged to obtain the multiple semantic units.
14. The method according to claim 13, characterized in that, The process of merging the initial text segments according to preset semantic integrity rules to obtain the plurality of semantic units includes: For each of the initial text segments, detect whether the initial text segment contains a negative word from a preset negative word library; If the initial text segment contains a negative word, then determine whether the initial text segment and the subsequent initial text segment satisfy the conditions of subject agreement or subject omission; If the conditions are met, the initial text segment is merged with the next initial text segment to obtain a semantic unit.
15. The method according to claim 13, characterized in that, The process of merging the initial text segments according to preset semantic integrity rules to obtain the plurality of semantic units includes: For each initial text segment, it is detected whether the initial text segment meets the preset condition triggering condition; wherein, the preset condition triggering condition includes: the initial text segment begins with a word in the preset condition-guided lexicon and the semantics are incomplete, or the initial text segment ends with a comma and the semantics are incomplete; If the initial text segment meets the preset trigger condition, then it is detected whether the next initial text segment contains a word from the preset result dictionary. If included, the initial text segment is merged with the subsequent initial text segment to obtain a semantic unit.
16. The method according to claim 13, characterized in that, The process of merging the initial text segments according to preset semantic integrity rules to obtain the plurality of semantic units includes: For each of the initial text segments, detect whether the initial text segment contains words from a preset enumerated guide dictionary; If the initial text segment contains words from the preset enumeration guide vocabulary, then the initial text segment is determined as the enumeration starting segment; Starting from the enumeration start segment, sequentially check whether the next initial text segment meets the preset enumeration end condition; The enumeration starting segment and all subsequent initial text segments up to and including the preset enumeration ending condition are merged to obtain a semantic unit.
17. The method according to claim 1, characterized in that, The step of generating input content that meets the input requirements of the target recognition model based on the target semantic unit includes: Obtain the original semantic position of each target semantic unit in the text to be identified; According to the order of the original semantic position representations, the target semantic units are sorted to obtain a sorted sequence of target semantic units; Using a preset connector, each target semantic unit in the target semantic unit sequence is connected sequentially to obtain the connected text; The input content is obtained by encapsulating the connection text using a preset prompt template.
18. A text processing device, characterized in that, include: The semantic unit acquisition module is used to acquire the text to be identified, perform semantic segmentation on the text to be identified, and obtain multiple semantic units; wherein, each semantic unit expresses an independent and complete semantic meaning; A semantic unit filtering module is used to filter out at least one target semantic unit from the plurality of semantic units; The input content generation module is used to generate target input content based on the target semantic unit; wherein the target input content is content that meets the input requirements of the target recognition model and is intended for target task recognition by the target recognition model.
19. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-17.
20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-17.
21. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-17.
