Text marking method and device based on large language model and keyword matching

Abstract

The invention discloses a text marking method and device based on large language model and keyword matching. The method comprises the following steps: constructing a standard label structure library; segmenting a to-be-labeled text into a plurality of text segments, and matching corresponding label paths in the standard label structure library; determining a label matching strategy according to the number of enumeration values in an enumeration value list under the label path matched with the text fragment; the text fragments with the label matching strategy being large model matching are stored in a task queue, and the task queue is packaged into a management and control configuration object; performing asynchronous parallel scheduling on the text fragments in the task queue based on the management and control configuration object to obtain a matching result of large language model matching; and combining the matching results to obtain a label list. The problems that in the prior art, a text labeling method based on a large language model is high in labeling cost and low in efficiency in a large-scale text scene are solved. Structured processing of mass text data is achieved, and the calculation cost and the processing efficiency of overall labeling are remarkably reduced.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a text tagging method and apparatus based on large language models and keyword matching. Background Technology

[0002] With the full arrival of the big data era, text data is experiencing explosive growth, its scale leaping from TB to PB or even EB. The efficient management and value extraction of this massive amount of text data (such as social media content, news information, and corporate documents) has become a core requirement for digital transformation across various fields. Against this backdrop, the scientific classification and tagging of text data is not only fundamental to achieving data structuring and improving information retrieval efficiency, but also a key technological link supporting subsequent data analysis, knowledge graph construction, and intelligent decision-making.

[0003] In recent years, Large Language Models (LLMs) have made groundbreaking progress in the field of natural language processing. Leveraging their massive parameter scale and training on vast corpora, they possess powerful contextual understanding, semantic mining, and content generation capabilities, leading to their widespread application in automatic text labeling tasks. Their core implementation logic involves taking text data and a label system description as input prompts, and then using the inference capabilities of the large language model to directly output the corresponding labels. Some advanced solutions incorporate a small amount of labeled data for fine-tuning to adapt to the labeling requirements of specific domains.

[0004] However, the inference process of large language models heavily relies on dedicated acceleration hardware such as GPUs. In actual deployment, multiple hardware constraints (insufficient GPU memory, insufficient computing power, PCIe / NVLink bandwidth bottlenecks, kernel scheduling and multi-process conflicts, model loading and cold start overhead) directly increase single inference latency and reduce batch throughput. In large-scale text and high-concurrency scenarios, the system is limited by the number of available GPUs and concurrency quotas, as well as memory and cache management strategies, making it difficult to linearly scale and stably maintain inference quality. Furthermore, mainstream large language models generally have input length limitations (e.g., GPT-3.5 has an input limit of 4096 tokens). For long texts such as novels, reports, and papers, segmentation and merging of results are required, increasing process complexity. At the same time, the computational load required for long text inference increases exponentially, leading to a significant increase in inference time. For example, the inference time for processing 10,000-word text can be dozens of times longer than that for a single sentence, significantly reducing overall processing efficiency. In addition, large language models have an inherent "illusion" defect, where the generated labels may output content that seems to conform to semantic logic but is actually inconsistent with the facts or deviates from the preset label system. Summary of the Invention

[0005] This application provides a text labeling method and apparatus based on large language models and keyword matching, which solves the problems of high labeling cost and low efficiency of existing text labeling methods based on large language models in large-scale text scenarios.

[0006] In a first aspect, embodiments of this application provide a text tagging method based on large language model and keyword matching, comprising: constructing a standardized tag structure library; wherein the standardized tag structure library includes multiple tag paths and their corresponding enumeration value lists; segmenting the text to be tagged into multiple text fragments, and matching the corresponding tag paths in the standardized tag structure library based on the text fragments; determining a tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment; wherein the tag matching strategy includes keyword matching and large language model matching; storing the text fragments whose tag matching strategy is large model matching into a task queue, and encapsulating them into a management configuration object; wherein the management configuration object includes the request payload, model information and system environment parameters corresponding to each text fragment; asynchronously and parallelly scheduling the text fragments in the task queue based on the management configuration object to obtain the matching result of large language model matching; merging the matching results of different tag matching strategies to obtain a tag list.

[0007] In conjunction with the first aspect, in one possible implementation, the construction of the standardized tag structure library includes: obtaining multi-level tags from a tag source and parsing the tag paths of the multi-level tags; determining the enumeration values ​​corresponding to each level of tags under the tag paths to form an enumeration value list; and establishing a mapping relationship between the tag paths and the corresponding enumeration value lists.

[0008] In conjunction with the first aspect, in one possible implementation, the construction of the standardized tag structure library includes: obtaining multiple multi-level tags from a tag source and parsing the tag paths of the multi-level tags; defining tag path IDs for each tag path; determining the enumeration values ​​corresponding to each level of tags under each tag path to form an enumeration value list; establishing a binding relationship between the tag path IDs and the corresponding enumeration value lists; and constructing a mapping relationship between the tag paths and the corresponding binding relationships.

[0009] In conjunction with the first aspect, in one possible implementation, determining the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment includes: if the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment is greater than a preset threshold, then keyword matching is used as the tag matching strategy; if the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment is not greater than the preset threshold, then large language model matching is used as the tag matching strategy.

[0010] In conjunction with the first aspect, in one possible implementation, before storing the text fragments matched by the tag matching strategy for the large model into the task queue, the method further includes: verifying the validity of the text fragments, including: checking whether the text fragments are not empty or contain garbled characters; and / or, determining whether the enumeration value list corresponding to the text fragments is not empty; and / or, verifying whether the mapping relationship between the enumeration values ​​corresponding to the text fragments and their tag paths is consistent with the standard tag structure library; adding valid text fragments to the task queue, and writing invalid text fragments to the exception log.

[0011] In conjunction with the first aspect, in one possible implementation, after storing the text fragments matched by the tag matching strategy for the large model into the task queue, the process includes: the large language model reading the model parameters corresponding to the model identifier from the system configuration and constructing prompt words; wherein, the model parameters include the interface address and access key corresponding to the model identifier; generating request payloads for each text fragment in the task queue; wherein, the request payloads include the content of the text fragment, the enumeration value list corresponding to the text fragment, the task identifier, and the prompt words constructed based on the text fragment; and determining system environment parameters according to the system configuration; wherein, the system environment parameters include concurrent request volume, timeout configuration parameters, and retry strategy parameters.

[0012] In conjunction with the first aspect, in one possible implementation, the asynchronous parallel scheduling of text segments in the task queue based on the management configuration object includes: obtaining system environment parameters and timeout parameters; wherein, the environment parameters include connection limit and timeout time; initializing the client and binding the system environment parameters and the timeout parameters; and initiating a call to the management configuration object using the client based on a tiered retry strategy; wherein, the tiered retry strategy includes: when the client captures an exception in the task, classifying the exception; the exception classification includes timeout exceptions, client errors, server errors, and general exceptions; when a timeout exception occurs, retrying based on the retry strategy until the retry is successful, or the maximum number of retries is reached, marking the task as failed and generating an empty result; when the... When a client error occurs, the error is recorded and retrying is stopped. When a server error occurs, if it is a temporary error, retrying is performed according to the retry strategy until the retry is successful, or the maximum number of retries is reached, the task is marked as failed and an empty result is generated. When a general exception occurs, a warning is recorded, and retrying is performed according to the retry strategy until the retry is successful, or the maximum number of retries is reached, the task is marked as failed and an empty result is generated. The retry strategy includes: retrying after waiting for a first time, and the waiting time for each subsequent retry increases based on the first time; when the client's request is completed, the session is forcibly closed and connection resources are released to prevent memory leaks, and random jitter is added during the waiting time to avoid blocking; when the maximum number of retries is reached, a fast failure mechanism is used to prevent a single task from blocking the entire queue.

[0013] In conjunction with the first aspect, in one possible implementation, before determining the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment, the method further includes: receiving a request submitted by the client and parsing the request parameters; validating the request parameters, returning an error frame and terminating if the validation fails, and standardizing the request object if the validation passes; obtaining a standardized tag structure library and text fragments based on the request object and request parameters, and matching tag paths for each text fragment in the standardized tag structure library; and generating independent subtasks based on each text fragment and its corresponding tag path.

[0014] In conjunction with the first aspect, in one possible implementation, after obtaining the tag list, the method further includes: based on the SSE protocol, the client updates the data in real time according to the event frame type, including: when the event frame is an initialization frame, sending request parameters to the client at the start of the session; when the event frame is a progress frame, sending task parameters to the client after each subtask is completed; wherein, the task parameters include frame type, frame identifier, timestamp, current number of processed tasks and total number of tasks, current incremental result set, full deduplication result set and task result flag; when the event frame is an exception frame, sending an exception description and session termination signal to the client; and when the event frame is a progress frame, sending the tag list to the client.

[0015] Secondly, embodiments of this application provide a text tagging device based on large language model and keyword matching, comprising: a construction module for constructing a standardized tag structure library; wherein the standardized tag structure library includes multiple tag paths and their corresponding enumeration value lists; a matching module for segmenting the text to be tagged into multiple text fragments, and matching the corresponding tag paths in the standardized tag structure library based on the text fragments; a determination module for determining a tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment; wherein the tag matching strategy includes keyword matching and large language model matching; an encapsulation module for storing the text fragments whose tag matching strategy is large model matching into a task queue, and encapsulating them into a management configuration object; wherein the management configuration object includes the request payload, model information and system environment parameters corresponding to each text fragment; a scheduling module for asynchronously and parallelly scheduling the text fragments in the task queue based on the management configuration object to obtain the matching result of large language model matching; and a merging module for merging the matching results of different tag matching strategies to obtain a tag list.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application's embodiments, by constructing a standardized tag structure library, provide a unified and structured tag system to support text tagging, ensuring the standardization and consistency of tags. By segmenting the text to be tagged into fragments and matching tag paths, the association between text fragments and the tag system is achieved. The tag matching strategy is determined based on the number of enumeration values. This approach leverages the efficiency of keyword matching to reduce computational costs when there are many enumeration values, while utilizing the semantic understanding capabilities of large language models to improve tag accuracy when there are few enumeration values, achieving a balance between effective cost and accuracy and enhancing user experience. Through the encapsulation of configuration objects and an asynchronous parallel scheduling mechanism, the throughput of large-scale text processing is effectively improved, alleviating the hardware resource constraints of large language model inference. This effectively solves the problem of high cost and low efficiency in existing text annotation methods based on large language models in large-scale text scenarios. Furthermore, it enables structured processing of massive amounts of text data, significantly reducing the overall computational cost and processing efficiency of tagging. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the text tagging method based on large language model and keyword matching provided in this application embodiment; Figure 2 A schematic diagram of the structure of a text tagging device based on a large language model and keyword matching provided in an embodiment of this application; Figure 3 This is a schematic diagram of the interactive timing for streaming output provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 This is a flowchart of a text tagging method based on a large language model and keyword matching provided in an embodiment of this application, including steps 101 to 106. Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order of the text tagging method based on large language models and keyword matching. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0022] Step 101: Construct a standardized tag structure library; wherein, the standardized tag structure library includes multiple tag paths and their corresponding enumeration value lists. In this embodiment, multi-level tags are obtained from the tag source, and the tag paths of the multi-level tags are parsed; the enumeration values ​​corresponding to each level of tags under the tag path are determined to form an enumeration value list; and a mapping relationship between the tag path and the corresponding enumeration value list is established.

[0023] Specifically, tag sources can be various types of data sources, such as industry standards, internal business rule manuals, and publicly available classification dictionaries. Through in-depth analysis and processing of these tag sources, multi-level tag information can be extracted.

[0024] When parsing tag paths in a multi-level tag hierarchy, it's necessary to clarify the hierarchical relationships between tags at each level. A tree structure can be used to represent this relationship, where each node represents a tag, and the path from the root node to a leaf node constitutes a tag path. For each level of tag within a tag path, its corresponding enumeration value is determined. Enumeration values ​​are the possible specific values ​​a tag can take; they can be predefined text, numbers, or other forms of identifiers.

[0025] Establishing a mapping between label paths and their corresponding enumeration value lists is a crucial step. This mapping allows for the quick and accurate identification of all possible enumeration values ​​for each label path. This is essential in subsequent text tagging processes, where, after matching a text fragment to a corresponding label path, the enumeration value list for that label path can be quickly determined, thus providing a basis for selecting an appropriate label matching strategy.

[0026] In practice, databases or data structures (such as hash tables) can be used to store these mapping relationships to improve query efficiency. Meanwhile, to ensure the integrity and accuracy of the standardized tag structure library, it is necessary to regularly check and update the constructed library to adapt to changes in business needs and updates to the tag source.

[0027] In this embodiment of the application, constructing a standardized tag structure library includes: obtaining multiple multi-level tags from a tag source and parsing the tag paths of the multi-level tags; defining tag path IDs for each tag path; determining the enumeration values ​​corresponding to each level of tags under each tag path to form an enumeration value list; establishing a binding relationship between tag path IDs and the corresponding enumeration value lists; and constructing a mapping relationship between tag paths and their corresponding binding relationships.

[0028] Specifically, in the process of acquiring multiple multi-level tags, the tag sources need to be comprehensively and meticulously screened. Different tag sources may have different formats and data quality, requiring preprocessing. For example, tags obtained from public data sources may have incomplete data or non-standard formats, requiring cleaning and conversion.

[0029] When parsing tag paths in a multi-level tag hierarchy, in addition to using a tree structure to represent membership relationships, a graph structure can also be used to handle more complex tag relationships. Some tags may have many-to-many relationships, which a tree structure may not be able to fully and accurately represent, while a graph structure can handle this situation better. For example, in a knowledge graph tag system, a topic tag may be associated with multiple domain tags; a graph structure can clearly display these complex relationships.

[0030] When defining tag path IDs for each tag path, it's crucial to ensure their uniqueness and scalability. Tag path IDs can be generated using hash algorithms or auto-incrementing numbering. Hash algorithms generate unique IDs based on the tag path content, but hash collisions may occur, requiring appropriate handling. Auto-incrementing numbering is simple and intuitive, but when dealing with a large number of tag paths, the range and management of the numbers may need to be considered. Simultaneously, tag path IDs should be readable to facilitate subsequent maintenance and management.

[0031] When determining the enumeration values ​​corresponding to each level of tags under each tag path, the semantics and business context of the tags should be fully considered. The definition of the enumeration values ​​should be accurate and comprehensive, covering all possible values ​​of the tags. For some ambiguous tags, further refinement and clarification may be necessary.

[0032] When establishing the binding relationship between tag path IDs and their corresponding tag paths, you can use a relational table or a nested data structure to store the information. A relational table clearly displays the correspondence between tag path IDs and tag paths, facilitating queries and statistics. A nested data structure allows you to directly embed tag paths within the tag path ID data structure, reducing data redundancy. At the same time, it's crucial to ensure the consistency and integrity of the binding relationship to avoid mismatches between tag path IDs and tag paths.

[0033] When constructing the mapping relationship between tag paths and their corresponding bindings (e.g., the mapping relationship of "tag path → {tag path ID, enumeration value list}"), data storage and query efficiency must be considered. Database indexing techniques can be used to accelerate query operations. For large-scale tag data, distributed storage and query methods can also be adopted to improve system performance and scalability. Furthermore, to ensure the accuracy of the mapping relationship, data validation and verification are necessary to ensure that the mapping between tag paths and binding relationships is correct. In practical applications, as business develops and changes, tag paths and enumeration value lists may be updated. An update mechanism can be established to update the mapping relationship in a timely manner to ensure the normal operation of the system.

[0034] Step 102: Segment the text to be tagged into multiple text fragments, and match the corresponding tag paths in the standardized tag structure library based on the text fragments. In this embodiment, word segmentation and sentence segmentation methods from natural language processing are used to segment the text to be tagged. For longer text to be tagged, it can be segmented into sentences first, and then each sentence can be further segmented into words to obtain multiple text fragments. For example, for a news report, it can be first segmented into multiple sentences according to punctuation marks such as periods, exclamation marks, and question marks, and then each sentence can be segmented into words or phrases using a word segmentation tool such as jieba to form text fragments.

[0035] After obtaining the text fragment, the corresponding tag path is matched against a standardized tag structure library. Various methods can be used, including string matching and semantic matching. String matching determines whether words in the text fragment are exactly or partially identical to words in the tag path in the standardized tag structure library. For example, if the text fragment contains "smartphone," and the standardized tag structure library contains the tag path "electronic products-phone-smartphone," then the text fragment can be considered to have matched this tag path. Semantic matching uses techniques such as word vectors and semantic similarity calculation to determine whether the text fragment and the tag path are semantically similar. For example, if the text fragment contains "mobile phone," although it is not exactly the same as the string "smartphone," semantic matching can reveal that they are semantically similar, and the text fragment can also be considered to have matched the tag path "electronic products-phone-smartphone."

[0036] For example, in step 101, this application constructs a complete standardized label structure library for the enumeration scenario, and concatenates each level of the label path with the enumeration value using the path separator "-" to obtain the complete label path, ensuring that the input format of subsequent matching is consistent.

[0037] During the matching process, a text fragment may match multiple tag paths. In this case, the paths can be sorted according to their matching strength, and the tag path with the highest matching strength should be selected. The matching strength can be determined by a combination of factors such as the number of matched words and semantic similarity. For example, a text fragment "Huawei smartphone" may match both "electronic products-phone-smartphone" and "electronic products-Huawei products-Huawei smartphone". Obviously, the matching strength of "electronic products-Huawei products-Huawei smartphone" is higher, and this tag path should be selected.

[0038] Meanwhile, to improve matching accuracy and efficiency, the standardized tag structure library can be optimized. For example, indexing tag paths and creating inverted indexes can quickly locate potentially matching tag paths during matching, reducing unnecessary traversal and comparisons. Furthermore, tag paths can be categorized and clustered, grouping similar tag paths together. During matching, category matching is performed first, narrowing the matching range and improving efficiency.

[0039] Step 103: Determine the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path of the text fragment matching; wherein, the tag matching strategy includes keyword matching and large language model matching. In this embodiment, if the number of enumeration values ​​in the enumeration value list under the tag path of the text fragment matching is greater than a preset threshold, then keyword matching is used as the tag matching strategy; if the number of enumeration values ​​in the enumeration value list under the tag path of the text fragment matching is not greater than the preset threshold, then large language model matching is used as the tag matching strategy.

[0040] Specifically, the preset threshold needs to be adjusted based on the actual application scenario and data characteristics. In scenarios where high tag accuracy is required and the number of enumerated values ​​is relatively small, the preset threshold can be set lower; while in scenarios with large amounts of data and abundant enumerated values, the preset threshold can be appropriately increased. For example, in a small enterprise knowledge base tagging system, since the number of tag enumerated values ​​is relatively limited, the preset threshold can be set to 10; while in a large internet content classification tagging system, the preset threshold may be increased to 50 or even higher.

[0041] When the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment exceeds a preset threshold, keyword matching is used as the tag matching strategy. Keyword matching is a matching method based on the literal content of the text. It determines the tag by searching for content in the text to be tagged that is exactly the same or partially the same as the keywords in the enumeration value list. The advantage of this method is that it is fast in calculation, simple to implement, and can process a large amount of data in a short time. For example, in a news information tagging system, for the "ball sports" tag path under the "sports" category, its enumeration value list contains many ball sports names, such as "basketball," "football," "volleyball," and "tennis." When processing a news article about a basketball game, keyword matching can quickly match the news text with the tag "ball sports - basketball" simply by finding the keyword "basketball" in the news text.

[0042] To improve the accuracy of keyword matching, the keywords can be preprocessed, such as removing stop words, performing stemming, etc. Stop words refer to words that frequently appear in the text but do not substantially help in semantic understanding, such as "of", "is", "in", etc. Removing stop words can reduce unnecessary interference. Stemming is to restore words to their stem forms. For example, restoring "running" to "run", which can expand the matching scope and improve the recall rate of matching. At the same time, a thesaurus of keywords can be established so that when words synonymous with the enumerated value keywords appear in the text fragment, matching can also be achieved. For example, synonyms of "football" include "soccer", "绿茵场运动" (which should be translated as "green field sport" or a more accurate equivalent if available), etc. When these synonyms appear in the text, the text fragment can also be matched with the "Ball Games - Football" label.

[0043] When the number of enumerated values in the enumerated value list under the label path matched by the text fragment is not greater than the preset threshold, the large language model matching is adopted as the label matching strategy. The large language model has powerful semantic understanding and reasoning capabilities and can handle complex language expressions and semantic relationships. It can perform semantic analysis on the text fragment and the enumerated values in the enumerated value list to judge the semantic similarity between them.

[0044] When using the large language model matching, the large language model needs to be fine-tuned to adapt to the specific label task. The large language model can be trained using the labeled sample data so that the large language model learns the semantic features of the label path and the enumerated values. At the same time, to improve the matching efficiency, techniques such as model compression and quantization can be adopted to reduce the computational amount and storage space of the large language model. In addition, multi-modal information, such as pictures, audio, etc., can be combined to further improve the matching accuracy of the large language model. For example, for the label task of painting works, in addition to the text description, the picture information of the painting work can also be input into the model, allowing the model to comprehensively consider the features of the text and the image and make a more accurate label judgment.

[0045] Before determining the label matching strategy according to the number of enumerated values in the enumerated value list under the label path matched by the text fragment, it also includes: receiving the request submitted by the client and parsing the request parameters; validating the request parameters, and when the validation fails, returning an error frame and terminating, and when the validation passes, standardizing the request object; based on the request object and the request parameters, obtaining the canonical label structure library and the text fragment, and matching the label path for each text fragment in the canonical label structure library; generating independent subtasks based on each text fragment and its corresponding label path.

[0046] Specifically, when receiving requests submitted by clients and parsing request parameters, it is necessary to accurately identify and extract different types of request parameters. For complex request parameters, which may have nested structures, a recursive algorithm must be used for parsing. For example, if the request parameters contain a list, and each element in the list is a dictionary containing multiple key-value pairs, then these nested data need to be processed recursively one by one.

[0047] Validating request parameters is a crucial step in subsequent processing. Validation includes the parameter type, value range, and format. For date parameters, check if they conform to a specific date format, such as "YYYY-MM-DD"; for numeric parameters, check if they are within a reasonable value range. If the request parameters include an age parameter, ensure its value is between 0 and 120. If validation fails, the returned error frame should contain detailed error information to facilitate client debugging and correction. Error information can include error codes and descriptions, such as error code "001" indicating "incorrect parameter type" and error description "age parameter should be an integer." After successful validation, standardizing the request object is essential for unifying the data format and structure, facilitating subsequent processing. The request object can be converted to a specific JSON or XML format to ensure data consistency. For example, converting the request object to JSON format facilitates data exchange between different systems.

[0048] When retrieving the canonical tag structure library and text fragments based on the request object and request parameters, the real-time nature and accuracy of the data must be considered. The canonical tag structure library may be continuously updated as business evolves, so it's crucial to ensure that the latest version is obtained. For text fragment retrieval, filtering and extraction should be performed according to the specific requirements in the request parameters. If the request parameters specify the source and scope of the text fragments, the fragments should be accurately retrieved from the corresponding data source.

[0049] When matching tag paths for each text fragment in the canonical tag structure library, multi-threading or distributed computing can be employed to improve matching efficiency. Generating independent subtasks based on each text fragment and its corresponding tag path facilitates subsequent parallel processing and task management. Each subtask can contain the text fragment, the matched tag path, and other relevant metadata. Task queues can be used to manage these subtasks, ensuring that tasks are processed in a specific order. For example, Redis can be used as a task queue, adding generated subtasks to the queue, and then multiple worker threads or processes retrieving and processing the subtasks from the queue.

[0050] Step 104: Store text fragments with a tag matching strategy of matching large models into the task queue and encapsulate them into a management configuration object; wherein, the management configuration object includes the request payload, model information and system environment parameters corresponding to each text fragment. In this embodiment of the application, before storing text fragments with a tag matching strategy of matching large models into the task queue, the method further includes: verifying the validity of the text fragments, including: checking whether the text fragments are not empty and do not contain garbled characters; and / or, determining whether the enumeration value list corresponding to the text fragments is not empty; and / or, verifying whether the mapping relationship between the enumeration values ​​corresponding to the text fragments and their tag paths is consistent with the standard tag structure library; adding valid text fragments to the task queue and writing invalid text fragments to the exception log.

[0051] Specifically, regular expressions can be used to check whether a text fragment is non-empty or free of garbled characters. For text fragments stored in encoding formats such as UTF-8, regular expressions can accurately determine whether there are garbled characters in the text fragment, and at the same time determine whether the text fragment is an empty string. If the text fragment contains garbled characters or is empty, it is determined to be an invalid text fragment.

[0052] Determining whether the enumeration value list corresponding to a text fragment (i.e., the list of enumeration values ​​matched by the text fragment from the tag path matched in the canonical tag structure library) is not empty requires checking the enumeration value list under the tag path matched by the text fragment. If the enumeration value list is empty, it means that this text fragment has no corresponding enumeration value and cannot perform a valid tag match, so it is judged as invalid text. For example, if a text fragment matches a specific tag path, but the enumeration value list under that tag path is empty, then this text fragment does not meet the requirements.

[0053] To verify whether the mapping relationship between the enumeration value and the tag path corresponding to the text fragment is consistent with the standard tag structure library, the mapping relationship between the enumeration value and the tag path corresponding to the text fragment must be compared with the mapping relationship in the standard tag structure library. If the two are inconsistent, it indicates that there is an error in the mapping relationship, and the text fragment will be judged as invalid.

[0054] When adding valid text fragments to a task queue, it's crucial to ensure the queue's stability and reliability. Message queue middleware like RabbitMQ can be used to implement this, offering high availability, scalability, and message persistence. Before adding a text fragment to the task queue, it needs to be encapsulated to include key information such as the text fragment itself, its corresponding enumeration value, and tag path. Furthermore, each task should be assigned a unique task identifier for easy tracking and management later.

[0055] When writing invalid text fragments to the exception log, the log must include detailed error information. In addition to the text fragment itself, the specific reason for the invalidity should be recorded, such as "text content is empty," "enumeration value list is empty," or "mapping relationship is inconsistent." Furthermore, information such as the time of the exception, the user or client involved, can be recorded for subsequent troubleshooting and repair. The exception log should be analyzed regularly to identify common problems causing invalid text fragments and to optimize and improve the system. For example, if a large number of text fragments are found to be invalid due to inconsistent mapping relationships, the update status of the specification tag structure library or the existence of problems with the matching rules should be checked.

[0056] In this embodiment of the application, after storing the text fragments for large model matching with the tag matching strategy into the task queue, the process includes: the large language model reading the model parameters corresponding to the model identifier from the system configuration and constructing prompt words; wherein, the model parameters include the interface address and access key corresponding to the model identifier; generating request payloads for each text fragment in the task queue; wherein, the request payload includes the content of the text fragment, the enumeration value list corresponding to the text fragment, the task identifier, and the prompt words constructed based on the text fragment; and determining system environment parameters according to the system configuration; wherein, the system environment parameters include concurrent request volume, timeout configuration parameters, and retry strategy parameters.

[0057] Specifically, when generating request payloads for each text fragment in the task queue, the integrity and accuracy of the request payloads must be ensured. The content of the text fragments must be preserved as is to ensure that the large language model can obtain complete text information. The enumeration value list corresponding to the text fragments must be accurate and can be extracted from the standardized label structure library. The task identifier is a unique identifier for each task, facilitating task tracking and management. The determination of prompt words should be based on the characteristics of the enumeration value list and the requirements of the large language model; the prompt words should guide the large language model to perform label matching more accurately.

[0058] When determining system environment parameters based on system configuration, the concurrent request volume setting should be determined based on the system's hardware resources and performance. If the system has strong processing capabilities, the concurrent request volume can be appropriately increased to improve processing efficiency; conversely, the concurrent request volume should be reduced to avoid system overload. The timeout configuration parameter refers to the condition that a request is considered timed out if no response is received within a specified time after it has been sent. The timeout configuration parameter setting should consider factors such as network latency and the processing time of large language models; a reasonable timeout configuration can prevent tasks from waiting for extended periods. The retry policy parameter specifies the number of retries and the retry interval when a request fails. If a request fails due to network fluctuations or other reasons, it can be retried according to the tiered retry strategy of this application to improve the success rate of the request.

[0059] When determining the model parameters of the large language model corresponding to each text fragment, the model identifier is used to uniquely identify the large language model used. Different large language models may have different performance and characteristics; the appropriate large language model should be selected based on the specific task requirements. The interface address is the network address for accessing the large language model; the correctness and availability of this address must be ensured. The access key is information used to verify access permissions. When using the large language model, the interface of the large language model must be correctly invoked based on the model identifier, interface address, and access key to achieve the tag matching function.

[0060] Step 105: Asynchronously and parallelly schedule text segments in the task queue based on the management configuration object to obtain the matching result of the large language model matching. In this embodiment, the system environment parameters and timeout parameters are obtained; the environment parameters include the connection limit and timeout time; the client is initialized and bound to the system environment parameters and timeout parameters; based on the hierarchical retry strategy, the client initiates a call to the management configuration object; the hierarchical retry strategy includes: when the client catches an exception in the task, the exception is classified; the exception classification includes timeout exception, client error, server error and general exception; when a timeout exception occurs, retry is performed based on the retry strategy until the retry is successful, or the maximum number of retries is reached and the task is marked as failed and an empty result is generated; when a client error occurs, the error is recorded and the retry is stopped; when a server error occurs, ... If the error is temporary, retry is performed according to the retry strategy until the retry succeeds, or the maximum number of retries is reached and the task is marked as failed, generating an empty result. When a general exception occurs, a warning is recorded, and retry is performed according to the retry strategy until the retry succeeds, or the maximum number of retries is reached and the task is marked as failed, generating an empty result. The retry strategy includes: retrying after waiting for the first time, with the waiting time for each subsequent retry increasing from the first time; forcibly closing the session and releasing connection resources after the client's request is completed to prevent memory leaks, and adding random jitter during the waiting time to avoid blocking; when the maximum number of retries is reached, a fast failure mechanism is used to prevent a single task from blocking the entire queue.

[0061] Specifically, when obtaining system environment parameters and timeout parameters, parameters such as the connection limit and timeout duration can be read from the system configuration file. Simultaneously, the validity of the obtained system environment parameters and timeout duration must be checked to ensure their values ​​are within a reasonable range. For example, the connection limit cannot be negative, and the timeout duration cannot be set too short or too long. During client initialization, parameters such as the connection limit and timeout duration can be passed to the client through the client's configuration interface, causing it to adhere to these settings during operation.

[0062] Based on a tiered retry strategy, when a client initiates a call to a management configuration object, exceptions need to be categorized. When a client catches an exception in a task, it must be accurately judged based on the exception's characteristics and type. For example, timeout exceptions are typically characterized by the task failing to complete within the specified timeout period; client errors are generally related to problems with the requesting client itself, such as incorrect parameter formats or insufficient access permissions; server errors are caused by problems on the server side, such as server overload or internal failures; and general exceptions are exceptions that cannot be clearly classified into other types.

[0063] When a timeout occurs, retrying according to the retry strategy aims to improve the success rate of the task. Waiting for the initial retry provides the system with a buffer time to resolve potential timeout issues, such as network congestion. Subsequent retry times increase from the initial timeout to avoid excessively frequent retries that could overload the system. Adding random jitter to the waiting time effectively prevents blocking caused by multiple tasks retrying simultaneously. For example, assuming the initial timeout is set to 1 second, with subsequent increments of 0.5 seconds and a random jitter range of ±0.2 seconds, the waiting time for the first retry (i.e., the retry interval) might be 0.8-1.2 seconds, and the waiting time for the second retry might be between 1.3-1.7 seconds. Alternatively, the waiting time for each retry could be double the previous one. After reaching the maximum number of retries (e.g., 3), a fail-fast mechanism is used to mark the task as failed and generate an empty result. This prevents a single task from occupying resources for an extended period, avoids blocking the entire queue, and ensures overall system performance.

[0064] When a client-side error occurs, promptly logging the error and aborting retries is crucial to avoid unnecessary retries. Recording error information, such as request parameters and exception stack traces, facilitates subsequent troubleshooting and repair. For example, if the client-side error is caused by an incorrectly formatted field in the request parameters, record the specific value of that field and the correct format requirements so that developers can check and correct it.

[0065] When a server error occurs, determining whether it is temporary is crucial. Temporary errors are usually caused by short-term server stress or malfunctions, such as momentary server overload or brief network outages. In such cases, retrying based on a retry strategy may restore normal operation. You can determine if an error is temporary by analyzing error messages and checking the server status. For example, if the server returns the error code "503 Service Unavailable," it can be initially determined to be a temporary error, and a retry should be performed according to the retry strategy.

[0066] When a common exception occurs, log the warning and retry based on the retry strategy. Logging warnings alerts developers to unknown exceptions in the system that require further attention and handling. Retrying using the retry strategy may restore the task to normal execution. During the retry process, the rules of increasing wait time and incorporating random jitter should also be followed to improve the efficiency and success rate of retrying. Once the client's request is completed, forcibly close the session and release connection resources to effectively prevent memory leaks and ensure stable system operation. Releasing connection resources returns system resources to the system, allowing them to be used for other tasks in a timely manner, improving system resource utilization.

[0067] Step 106: Merge the matching results of different label matching strategies to obtain a label list. In this embodiment, multiple methods are used to merge the matching results of different label matching strategies. First, the data format of the results from the large model matching and the results from other label matching strategies must be unified to ensure that the fields and data types of all results are consistent, facilitating subsequent processing. A unified data structure can be defined to convert the matching results generated by different strategies into this structure.

[0068] Next, the matching results are deduplicated. Since different tag matching strategies may generate duplicate tags, it is necessary to compare them using their unique identifiers and remove duplicate tags.

[0069] Then, sort the results according to the label confidence level. Labels obtained from large model matching and other strategy matching may have different confidence values. The labels can be sorted from highest to lowest confidence level, with higher confidence labels listed first. This allows for priority consideration of higher confidence labels when using the label list.

[0070] When merging results, tag compatibility also needs to be considered. Some tags may be semantically similar or have an inclusion relationship. In this case, semantically similar tags can be merged, or tags with inclusion relationships can be integrated, retaining only the most representative tags.

[0071] After the merge is complete, the tag list needs to be validated. Check if the tag list conforms to the requirements of the standardized tag structure library, ensuring that the mapping between tag paths and enumeration values ​​is correct for all tags. Also, check the completeness of the tag list to see if any necessary tags are missing.

[0072] In this embodiment of the application, after obtaining the tag list, the method further includes: based on the SSE protocol, the client updates the data in real time according to the event frame type, including: when the event frame is an initialization frame, sending request parameters to the client at the start of the session; when the event frame is a progress frame, sending task parameters to the client after each subtask is completed; wherein, the task parameters include frame type, frame identifier, timestamp, current number of processed tasks and total number of tasks, current incremental result set, full deduplication result set and task result flag; when the event frame is an exception frame, sending an exception description and session termination signal to the client; when the event frame is a progress frame, sending the tag list to the client.

[0073] Specifically, when the event frame is an initialization frame, the sent request parameters include basic task information, such as the task type and the expected tag matching target, so that the client can clearly understand the task requirements. These parameters can be read from the task configuration file and validated as necessary to prevent task failure due to incorrect parameters.

[0074] When the event frame is a progress frame, the task parameters sent after each subtask is completed include frame type, frame identifier, timestamp, current number of processed tasks and total number of tasks, current incremental result set, full deduplication result set, and task result flag. The frame type clearly identifies the specific meaning of the progress frame, and the frame identifier, as a unique identifier for each progress frame, helps track task progress. The timestamp records the completion time of the subtask, facilitating subsequent analysis of task execution efficiency. The current number of processed tasks and the total number of tasks allow the client to understand the overall progress of the task in real time. The current incremental result set contains newly generated tag matching results after the completion of the subtask, while the full deduplication result set is the complete result after deduplication processing based on the incremental result set. The task result flag indicates whether the subtask was successfully completed. To ensure the real-time nature of the task parameters, asynchronous message queues and other technologies can be used to promptly deliver subtask completion messages to the client.

[0075] When an event frame is an abnormal frame (missing or invalid parameters), the sent exception description should include the specific type of exception, the location of the exception, and possible causes. A session termination signal should be sent promptly so that the client can stop related operations in time, avoiding resource waste. Simultaneously, the exception information should be logged for subsequent troubleshooting and repair.

[0076] When the event frame is a progress frame, the tag list can be checked again before sending it to the client to ensure its accuracy and completeness. If the tag list is large, it can be sent in batches to avoid network congestion caused by sending too much data at once. Furthermore, necessary explanatory information, such as the tag list generation time and the matching strategy used, can be attached along with the tag list to facilitate client processing and use.

[0077] like Figure 3 The diagram illustrates the interactive timing of the streaming output in this application. The entire processing chain is designed as a collaborative architecture of "entry layer—control layer—execution layer—SSE channel" to ensure that the entire process from request entry to result return can be recorded in real time and continuously output. As shown in the diagram, after a user initiates an interface request, the entry layer completes request parameter parsing and validation, and the control layer triggers streaming matching accordingly. The execution layer first prepares the standardized tag structure library and performs text slicing, then selects a matching strategy for each text fragment and tag path combination and executes the matching. Whenever keyword matching or large language model matching completes a subtask, the execution layer immediately returns the event data corresponding to that subtask. The control layer generates and outputs a progress frame based on this data, and the client receives "data: {...}" data in real time, gradually updating the streaming output data. After all subtasks are completed, the control layer generates a completion frame, which includes a tag list and statistical information composed of the deduplicated final matching results, and returns it to the client as a session termination signal.

[0078] The entry layer receives the JSON request payload submitted by the client. The payload includes the content of a text fragment, a list of corresponding enumeration values, a task identifier, and a prompt determined based on the enumeration value list. It parses the corresponding model identifier, interface address, access key, concurrent request volume, timeout configuration parameters, retry policy parameters, and other parameters, and performs validation on required parameters. If validation fails, the streaming protocol is not interrupted; instead, an error event frame is immediately generated and written to the response in text event stream format, allowing the client to receive an error description and session termination signal within the same long-lived connection. After successful validation, the entry layer hands over the standardized internal request object to the control layer, initiating the streaming matching process.

[0079] The control layer performs subsequent tasks based on the configuration parameters and policy constraints of the internal request object, including tag structure acquisition, text slicing, task queue construction, concurrency control, and result convergence. To adapt to streaming output, the control layer does not wait for all subtasks to complete but instead sends an event frame immediately upon completion of each subtask. The system continuously records "how many times it has been completed" and "how many times in total." Each time a new result is obtained, the newly added result and the deduplicated total result are summarized together, and the progress and cumulative matching results are sent frame by frame through the same long connection. When all subtasks are completed, the control layer summarizes the matching results and statistical information (including the number of slices, the number of times the policy is used, the timeout value, and the effectiveness of the concurrency threshold, etc.) and generates a completion frame as the final status output of this session.

[0080] The execution layer is used for data preparation and the execution of specific matching algorithms. Based on configuration items, the execution layer obtains the tag tree and parses the mapping between tag paths and tag path IDs. Then, it batches and populates enumeration values ​​according to the path, forming a mapping relationship of "tag path—tag path ID—enumeration value list". Text processing uses word segmentation and sentence segmentation methods from natural language processing to divide the text to be tagged. Each text segment is then paired with each tag path. Each pairing of a text segment and a tag path constitutes a subtask. These subtasks are organized into a task queue to be processed, and the number of enumeration values ​​determines the tag matching strategy. After each subtask is completed, the execution layer constructs a structured event data, including the newly added matching results and the deduplicated set up to date, and returns this data to the upstream. To ensure the reliability of the results, all matching results returned by the large language model undergo strict candidate set verification. Content not within the standardized tag structure library is filtered and a warning is recorded.

[0081] The SSE channel is a client-facing, one-way, real-time transmission channel implemented using HTTP persistent connections and a text event stream format. The server continuously writes event frames, each beginning with "data:" and containing a JSON object, separated by blank lines. The SSE channel connection remains open until all subtasks are completed. Event frames use a unified data structure, including frame type, frame identifier, timestamp, current number of processed tasks, total number of tasks, current incremental result set, full deduplication result set, and task result flags. A completion frame is output at the end of the session, along with processing completion status and statistics. This format allows clients to receive processing progress and results frame by frame, without waiting for the entire process to finish. To bridge internal asynchronous execution and external streaming writes, the system uses a generator to handle the event stream: internal asynchronous processing produces a structured result upon completion of each subtask, and the outer generator synchronously converts this into an event frame and writes it to the response stream. Concurrency control is achieved by limiting the number of concurrent requests, and together with timeout configuration parameters, the resource consumption and failure recovery path of large language model tasks are constrained, ensuring that continuous progress frames can still be stably produced in high-concurrency and long-duration scenarios.

[0082] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0083] like Figure 2 As shown in the figure, this application embodiment also provides a text tagging device 200 based on a large language model and keyword matching. The device includes: a construction module 201, a matching module 202, a determination module 203, an encapsulation module 204, a scheduling module 205, and a merging module 206, as detailed below.

[0084] Module 201 is used to build a standardized tag structure library. This library includes multiple tag paths and their corresponding enumerated value lists.

[0085] The matching module 202 is used to segment the text to be tagged into multiple text fragments and match the corresponding tag paths in the standard tag structure library based on the text fragments.

[0086] The determination module 203 is used to determine the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment. The tag matching strategy includes keyword matching and large language model matching.

[0087] The encapsulation module 204 stores text fragments with a tag matching strategy of matching large models into a task queue and encapsulates them into a management configuration object. The management configuration object includes the request payload, model information, and system environment parameters corresponding to each text fragment.

[0088] The scheduling module 205 is used to perform asynchronous parallel scheduling of text segments in the task queue based on the management configuration object, and obtain the matching results of the large language model matching.

[0089] The merging module 206 is used to merge the matching results of different tag matching strategies to obtain a tag list.

[0090] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0091] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0092] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, for example, as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. Memory controllers can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functions can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0093] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0094] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0095] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0096] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0097] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A text tagging method based on large language models and keyword matching, characterized in that, include: Construct a standardized tag structure library; wherein, the standardized tag structure library includes multiple tag paths and their corresponding enumeration value lists; The text to be tagged is divided into multiple text fragments, and the corresponding tag paths are matched in the standard tag structure library based on the text fragments; The tag matching strategy is determined based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment; wherein, the tag matching strategy includes keyword matching and large language model matching; The text segments matched by the tag matching strategy for the large model are stored in the task queue and encapsulated as a management configuration object; wherein, the management configuration object includes the request payload, model information and system environment parameters corresponding to each text segment; Based on the management configuration object, the text segments in the task queue are asynchronously and in parallel scheduled to obtain the matching results of the large language model matching. Merge the matching results from different tag matching strategies to obtain a tag list.

2. The method according to claim 1, characterized in that, The construction of the standardized tag structure library includes: Obtain multi-level tags from the tag source and parse the tag paths of the multi-level tags; Determine the enumeration values ​​corresponding to each level of the tag under the tag path, and construct an enumeration value list; Establish a mapping relationship between the label path and the corresponding enumeration value list.

3. The method according to claim 1, characterized in that, The construction of the standardized tag structure library includes: Obtain multiple multi-level tags from the tag source, and parse the tag paths of the multi-level tags; Define a tag path ID for each of the aforementioned tag paths; Determine the enumeration values ​​corresponding to each level of the tags under each of the aforementioned tag paths, and construct an enumeration value list; Establish a binding relationship between the tag path ID and the corresponding enumeration value list; Construct a mapping relationship between the tag path and the corresponding binding relationship.

4. The method according to claim 1, characterized in that, The step of determining the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment includes: If the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment is greater than the preset threshold, then keyword matching is used as the tag matching strategy. If the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment is not greater than the preset threshold, then the large language model matching is used as the tag matching strategy.

5. The method according to claim 1, characterized in that, Before storing the text segment matched by the tag matching strategy for the large model into the task queue, the method further includes: Verifying the validity of the text fragment includes: Check whether the text fragment is not empty and free of garbled characters; and / or, Determine whether the enumeration list corresponding to the text fragment is not empty; and / or, Verify whether the mapping relationship between the enumeration value corresponding to the text fragment and its tag path is consistent with the standard tag structure library; Add valid text fragments to the task queue and write invalid text fragments to the exception log.

6. The method according to claim 1, characterized in that, After storing the text segments that match the large model according to the label matching strategy into the task queue, the process includes: The large language model reads the model parameters corresponding to the model identifier from the system configuration and constructs prompt words; wherein, the model parameters include the interface address and access key corresponding to the model identifier; Generate a request payload for each text fragment in the task queue; wherein, the request payload includes the content of the text fragment, a list of enumerated values ​​corresponding to the text fragment, a task identifier, and a prompt word constructed based on the text fragment; The system environment parameters are determined based on the system configuration; wherein, the system environment parameters include concurrent request volume, timeout configuration parameters, and retry policy parameters.

7. The method according to claim 1, characterized in that, The asynchronous parallel scheduling of text segments in the task queue based on the management configuration object includes: Obtain the system's environment parameters and timeout parameters; wherein, the environment parameters include the connection limit and timeout period; Initialize the client and bind the system environment parameters and the timeout parameters; Based on a tiered retry strategy, the client initiates a call to the control configuration object. The tiered retry strategy includes: When the client catches an exception in a task, the exception is categorized; the exception categories include timeout exceptions, client errors, server errors, and general exceptions. When the timeout exception occurs, retry is performed based on the retry strategy until the retry is successful, or the maximum number of retries is reached and the task is marked as failed and an empty result is generated. When the aforementioned client error occurs, the error is logged and retry is aborted; When the server error occurs, if it is a temporary error, it will be retried according to the retry policy until the retry is successful, or the maximum number of retries is reached and the task is marked as failed and an empty result is generated. When the aforementioned common exception occurs, a warning is recorded, and a retry is performed based on the retry policy until the retry is successful, or the maximum number of retries is reached, the task is marked as failed and an empty result is generated. The retry strategy includes: retrying after waiting for a first time, with the waiting time for each subsequent retry increasing based on the first time; forcibly closing the session and releasing connection resources after the client's request is completed to prevent memory leaks; and adding random jitter during the waiting time to avoid blocking; and using a fast failure mechanism after the maximum number of retries is reached to prevent a single task from blocking the entire queue.

8. The method according to claim 1, characterized in that, Before determining the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment, the method further includes: Receive requests submitted by clients and parse the request parameters; The request parameters are validated. If the validation fails, an error frame is returned and the process is terminated. If the validation passes, the request object is standardized. Based on the request object and request parameters, obtain the standardized tag structure library and text fragments, and match the tag path for each text fragment in the standardized tag structure library; Each text fragment is used to generate an independent subtask based on its corresponding tag path.

9. The method according to claim 1, characterized in that, After obtaining the tag list, the process also includes: Based on the SSE protocol, the client performs real-time updates according to the event frame type, including: When the event frame is an initialization frame, request parameters are sent to the client at the start of the session; When the event frame is a progress frame, task parameters are sent to the client after each subtask is completed; wherein, the task parameters include frame type, frame identifier, timestamp, current number of processed tasks and total number of tasks, current incremental result set, full deduplication result set and task result flag; When the event frame is an abnormal frame, an abnormal description and a session termination signal are sent to the client. When the event frame is a progress frame, the tag list is sent to the client.

10. A text tagging device based on large language model and keyword matching for implementing the method described in any one of claims 1-9, characterized in that, include: A construction module is used to build a standardized tag structure library; wherein, the standardized tag structure library includes multiple tag paths and their corresponding enumeration value lists; The matching module is used to segment the text to be tagged into multiple text fragments and match the corresponding tag paths in the standard tag structure library based on the text fragments; The determination module is used to determine the tag matching strategy based on the number of enumeration values ​​in the enumeration value list under the tag path matched by the text fragment; wherein, the tag matching strategy includes keyword matching and large language model matching; An encapsulation module is used to store text fragments that match the large model according to the tag matching strategy into a task queue and encapsulate them into a management configuration object; wherein, the management configuration object includes the request payload, model information and system environment parameters corresponding to each text fragment; The scheduling module is used to asynchronously and in parallel schedule text segments in the task queue based on the management and control configuration object to obtain the matching results of the large language model matching. The merging module is used to merge the matching results of different tag matching strategies to obtain a tag list.

Images