Text labeling method, system, device and medium based on multi-model cooperation

Abstract

This invention relates to the field of artificial intelligence technology, specifically providing a text annotation method, system, device, and medium based on multi-model collaboration. The method includes: acquiring user-submitted text to be annotated, annotation requirements, and performance constraint parameters; parsing the annotation requirements to generate a standardized annotation task set and extracting text features; decomposing the tasks into atomic annotation tasks and constructing task execution paths based on dependencies; using a multi-attribute decision algorithm to match the optimal model for each atomic task based on text features, performance constraints, and a model profile library, and generating a scheduling plan according to the execution path; invoking multi-model collaborative inference to obtain the original annotation results, and outputting structured annotation results after fusion processing. This invention achieves dynamic decoupling between annotation tasks and models, supports multi-model collaborative inference and result verification, and significantly improves the flexibility, accuracy, and efficiency of text annotation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a text annotation method, system, device, and medium based on multi-model collaboration. Background Technology

[0002] Traditional text annotation systems generally employ a hard-coded pattern where one task type corresponds to one fixed model. Entity recognition tasks can only call a pre-set entity recognition model and cannot dynamically select the optimal model based on text complexity, domain characteristics, or system load.

[0003] This rigid binding leads to two drawbacks: on the one hand, using the same model to process simple texts and complex professional texts (such as legal documents) results in wasted computing power or insufficient accuracy; on the other hand, it is impossible to adaptively adjust the model selection strategy according to performance constraints (such as latency requirements and computing power limits), thus limiting the overall resource utilization and annotation efficiency. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a text annotation method, system, device and medium based on multi-model collaboration to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a text annotation method based on multi-model collaboration, comprising: Obtain the text to be annotated, annotation requirements, and performance constraint parameters submitted by the user; The text to be labeled and the labeling requirements are parsed to generate a standardized labeling task set, and the text features of the text to be labeled are extracted. The annotation tasks in the standardized annotation task set are decomposed into multiple atomic annotation tasks, and task execution paths are constructed based on the dependencies between the atomic annotation tasks. Based on the text features, the performance constraint parameters, and the pre-built model profile library, a multi-attribute decision algorithm is used to match the optimal annotation model for each atomic annotation task, and a model scheduling execution plan is generated based on the task execution path and the matched annotation model. According to the model scheduling execution plan, multiple annotation models are invoked to perform collaborative inference, the task annotation results output by each annotation model are obtained, and the multiple task annotation results are fused to generate and output the annotation results.

[0006] In one optional implementation, the user-submitted text to be annotated, annotation requirements, and performance constraint parameters are obtained, including: Receive user-uploaded datasets of text to be labeled; Receive annotation requests from users in natural language format; The annotation mode selected by the user is received and mapped to specific performance constraint parameters. The performance constraint parameters include at least one or more of the following: annotation accuracy priority, inference latency requirements, deployment environment limitations, computing power resource thresholds, and data compliance requirements.

[0007] In one optional implementation, the text to be annotated and the annotation requirements are parsed to generate a standardized annotation task set, including: Semantic parsing is performed on the annotation requirements to extract one or more explicit annotation task types and the corresponding tag system for each type; Obtain the attribute features of the text to be labeled, and infer one or more implicit labeling task types and the corresponding label system for each type based on the attribute features; The explicit annotation task type and the implicit annotation task type are merged to generate an initial annotation task set for the text to be annotated, wherein each annotation task is associated with the text to be annotated or a partial fragment thereof as an annotation object; The initial annotation task set is pushed to the user for confirmation, and a standardized annotation task set is generated based on the user feedback.

[0008] In one optional implementation, the attribute features of the text to be labeled are obtained, and one or more implicit labeling task types and corresponding tag systems for each type are inferred based on the attribute features, including: Extract the attribute features of the text to be labeled, wherein the attribute features include at least domain attributes and text length; The attribute features are encoded into feature vectors and input into a pre-trained TextCNN classifier, which includes convolutional layers, pooling layers, and fully connected layers. The TextCNN classifier outputs the probability distribution of each candidate latent labeling task, and determines the candidate labeling task type with a probability higher than a preset threshold as the latent labeling task type. Based on the determined implicit annotation task type, the label system corresponding to the task type is retrieved from the pre-built label system mapping table and used as the label system for the implicit annotation task.

[0009] In an optional implementation, the annotation tasks in the standardized annotation task set are decomposed into multiple atomic annotation tasks, and task execution paths are constructed based on the dependencies between the atomic annotation tasks, including: Based on a predefined atomic operation library, each annotation task is decomposed into indivisible atomic annotation tasks according to the principle of minimum indivisibility; The input dependencies of each atomic labeling task are analyzed. The input dependencies include: if the input parameters of one atomic labeling task depend on the output of another atomic labeling task, then it is determined that there is a strong serial dependency between the two. A directed acyclic graph is constructed based on the aforementioned dependencies, where nodes represent atomic labeling tasks and directed edges represent dependencies between tasks. The directed acyclic graph is topologically sorted to generate acyclic task execution sequences, and parallel execution groups are divided according to dependency levels, wherein atomic labeling tasks without prerequisite dependencies are assigned to the same parallel execution group.

[0010] In an optional implementation, based on the text features, the performance constraint parameters, and a pre-built model profile library, a multi-attribute decision algorithm is used to match the optimal annotation model for each atomic annotation task, and a model scheduling execution plan is generated based on the task execution path and the matched annotation model, including: For each atom annotation task, an initial set of models suitable for the task is selected from a pre-built model profile library; Hard constraints and soft constraints are generated using the performance constraint parameters, wherein the hard constraints are used to define the mandatory indicator boundaries that the model must meet, and the soft constraints are used to measure the degree to which the model meets user preference indicators. By applying the hard constraints, any model in the initial model set that does not meet the mandatory index boundary is eliminated, and the remaining models constitute the candidate model set. Using annotation accuracy, inference latency, computing power consumption, and domain adaptability as evaluation indicators, the TOPSIS multi-attribute decision algorithm is used to comprehensively score each model in the candidate model set. The comprehensive score is obtained by calculating the distance between each model and the ideal optimal model and the ideal worst model. The closer the model is to the ideal optimal model and the farther the model is from the ideal worst model, the higher the comprehensive score. In addition, when calculating the distance, a score penalty is imposed on the model that deviates from the user preference index according to the soft constraint condition. Combining the deep Q-network reinforcement learning algorithm, with the optimization objectives of improving labeling accuracy, reducing overall inference time, and improving resource utilization, any behavior that violates the hard or soft constraints is included as a negative reward in the reward function. The model selection strategy is dynamically adjusted through iterative learning to output the optimal matching model for each atomic labeling task. Based on the dependencies of each atomic annotation task in the task execution path, the matched models are arranged into a scheduling execution sequence according to the dependencies, and a model scheduling execution plan is generated.

[0011] In an optional implementation, before fusing the multiple task annotation results, the method further includes: For cases where multiple models output different annotation results for the same labeled object, the domain adaptation weight of each model in the task type to which the labeled object belongs and the confidence of the current output result are obtained. The domain adaptation weight and the confidence are weighted and calculated. The candidate results are sorted according to the weighted score, and one or more results with the highest weighted score are retained as the set of results to be verified. If the result with the highest weighted score is unique and its score is significantly higher than other results, then the result is directly determined as the result that passes the first verification. For each result in the set of results to be verified, a validity check is performed based on a predefined annotation rule base, and results that violate any rule are filtered out. The rules include at least the following: the subject and object in the relation triple must both exist in the entity list; the sentiment annotation result must be consistent with the sentiment tendency of the corresponding opinion evaluation word; and the type of the event element must match the predefined element type of the event. If at least one valid result still exists after rule verification, the result with the highest weighted score is selected as the result that passes the second verification. When the score difference between the highest weighted scores after the second verification is less than a preset threshold, or when all results are filtered by rules after the second verification, resulting in no valid results, the corresponding original text fragment, all conflicting task annotation results, and related annotation rules are input into the pre-trained large language model arbitrator, and the large language model outputs the final arbitration result. Based on the results of the first verification, the second verification, and the arbitration result determined after the third verification, output the verified task labeling results corresponding to each atomic labeling task.

[0012] Secondly, the present invention provides a text annotation system based on multi-model collaboration, comprising: The parameter acquisition module is used to acquire the text to be labeled, labeling requirements, and performance constraint parameters submitted by the user. The task generation module is used to parse the text to be labeled and the labeling requirements, generate a standardized labeling task set, and extract the text features of the text to be labeled. The task decomposition module is used to decompose the annotation tasks in the standardized annotation task set into multiple atomic annotation tasks, and construct task execution paths according to the dependencies between the atomic annotation tasks. The model scheduling module is used to match the optimal annotation model for each atomic annotation task based on the text features, the performance constraint parameters and the pre-built model profile library, using a multi-attribute decision algorithm, and to generate a model scheduling execution plan based on the task execution path and the matched annotation model. The result processing module is used to call multiple annotation models to perform collaborative inference according to the model scheduling execution plan, obtain the task annotation results output by each annotation model, and perform fusion processing on the multiple task annotation results to generate and output the annotation results.

[0013] Thirdly, a device is provided, comprising: Memory is used to store text annotation programs based on multi-model collaboration; A processor, configured to implement the steps of the multi-model collaborative text annotation method as provided in the first aspect when executing the multi-model collaborative text annotation program.

[0014] Fourthly, a computer-readable medium is provided, on which a multi-model collaborative text annotation program is stored, wherein when the multi-model collaborative text annotation program is executed by a processor, the multi-model collaborative text annotation method as provided in the first aspect is implemented.

[0015] The text annotation method, system, device, and medium based on multi-model collaboration provided by this invention have the following beneficial effects: First, this application constructs a pre-configured model profile library, recording the adaptability of each annotation model in dimensions such as text type and domain attributes, and uses a multi-attribute decision algorithm to dynamically match the optimal model for each atomic annotation task. Specifically, the system can select candidate models suitable for the current text characteristics from the model profile library based on the domain features (such as whether it is a legal document) and complexity of the text to be annotated, and comprehensively evaluate indicators such as accuracy, latency, and computing power consumption through algorithms such as TOPSIS. This allows for the selection of lightweight and efficient models for simple texts to save computing power, and the selection of high-precision domain models for complex and professional texts to ensure quality, achieving on-demand matching and overcoming the resource waste and accuracy defects of the traditional hard-binding mode.

[0016] Second, this application introduces user-submitted performance constraint parameters (such as inference latency limits and computing resource thresholds) during the scheduling process. These constraints serve as hard boundaries and soft penalties for model selection, while the scheduling strategy is dynamically optimized using a deep Q-network reinforcement learning algorithm. Specifically, when the system is in a high-concurrency or resource-constrained scenario, the scheduler prioritizes models that meet the latency and computing power requirements based on performance constraints. It also applies negative feedback to violations through a reinforcement learning reward function, thereby adaptively adjusting the model selection strategy. This avoids the efficiency bottleneck caused by the inability to flexibly switch models according to system load or business needs in traditional solutions, improving overall resource utilization and annotation throughput.

[0017] In summary, this application breaks the strong coupling between tasks and models through a collaborative mechanism that integrates model profiling library, multi-attribute decision scheduling, and performance constraints, achieving dynamic and adaptive matching between annotation tasks and model resources, and significantly improving the system's flexibility, accuracy, and efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic flowchart illustrating the method for generating a standardized annotation task set according to an embodiment of the present invention.

[0021] Figure 3 This is a schematic flowchart illustrating the disassembly and annotation task of a method according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic flowchart illustrating a method for triple verification of task annotation results according to an embodiment of the present invention.

[0023] Figure 5 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0027] The text annotation method based on multi-model collaboration provided in this embodiment of the invention is executed by a computer device, and correspondingly, the text annotation system based on multi-model collaboration runs on the computer device.

[0028] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a text annotation system based on multi-model collaboration. Depending on different needs, the order of steps in this flowchart can be changed, and some can be omitted.

[0029] like Figure 1 As shown, the method includes: S1. Obtain the text to be annotated, annotation requirements, and performance constraint parameters submitted by the user; S2. Analyze the text to be labeled and the labeling requirements, generate a standardized labeling task set, and extract the text features of the text to be labeled; S3. Decompose the annotation tasks in the standardized annotation task set into multiple atomic annotation tasks, and construct task execution paths according to the dependencies between the atomic annotation tasks; S4. Based on the text features, the performance constraint parameters, and the pre-built model profile library, a multi-attribute decision algorithm is used to match the optimal annotation model for each atomic annotation task, and a model scheduling execution plan is generated based on the task execution path and the matched annotation model. S5. According to the model scheduling execution plan, call multiple annotation models to perform collaborative inference, obtain the task annotation results output by each annotation model, and perform fusion processing on the multiple task annotation results to generate and output the annotation results.

[0030] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0031] S101. Receive the text dataset to be annotated uploaded by the user. Users can upload text files to be processed through the system's web interface or API interface. For example, a user uploads a batch of legal judgment documents, each in TXT format, containing paragraphs such as case facts, points of contention, and reasoning for the judgment. The system performs integrity verification and format checks on the uploaded files, and stores them in the processing queue after confirming that they are correct.

[0032] S102. Receive annotation requests input by users in natural language. Users do not need to follow a strict instruction format and can directly describe their annotation goals in the interface input boxes. For example, a user might input: "Please help me extract the plaintiff's name, defendant's name, cause of action, and judgment result from these court documents, and analyze whether the court's attitude towards the points of contention is supportive or dismissal." The system will temporarily store this natural language text as an annotation request instruction.

[0033] S103. Receive the annotation mode selected by the user and map the annotation mode to specific performance constraint parameters. The system provides multiple selectable modes on the user interface, such as "Fast Mode," "Balanced Mode," and "High Precision Mode." The user selects one according to the business scenario. Assuming the user's current requirement is high in real-time performance, "Fast Mode" is selected.

[0034] The system internally maintains a mapping logic from mode to performance parameters. When a user selects "Fast Mode," the system maps it to the following performance constraints: inference latency capped at 500 milliseconds, single-sample GPU memory usage capped at 2 gigabytes, annotation accuracy capped at 0.75, computing resource priority set to low priority, deployment environment allows the use of lightweight model instances, and data compliance requirements adhere to general privacy protection protocols. If the user selects "Balanced Mode," the mapping is: inference latency capped at 1000 milliseconds, GPU memory capped at 4 gigabytes, accuracy capped at 0.85, and computing resource priority set to medium. If the user selects "High Precision Mode," inference latency is unrestricted, GPU memory capped at 8 gigabytes, accuracy capped at 0.95, and computing resource priority set to high.

[0035] In addition, users can manually adjust any of the above parameters through advanced options. For example, if a user modifies the latency limit to 800 milliseconds on top of "Fast Mode," the system will accept this user-defined value, overriding the default mapping, and generating the final personalized performance constraint parameter set. These parameters will then be passed to the scheduler for hard constraint elimination (e.g., any model with an inference latency exceeding 500 milliseconds will be directly excluded) and soft penalty calculation (e.g., in the TOPSIS score, models with higher inference latency receive higher penalty coefficients).

[0036] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. Please refer to [the relevant documentation]. Figure 2 .

[0037] S201. Perform semantic parsing on the annotation requirements to extract explicit annotation task types and tag systems. Input the user's natural language annotation requirements into the fine-tuned large language model (such as LLaMA-3-70B). Construct the following Prompt template: "You are a text annotation task parsing assistant. Please extract the specific annotation task type from the following user requirements and provide the corresponding label system for each task type. User requirements: {User input}. Output format: Task type 1: [Label value 1, Label value 2, ...]; Task type 2: [...]." Taking the aforementioned user request, "Please help me extract the plaintiff's name, defendant's name, cause of action, and judgment result from these court documents, and analyze whether the court's attitude towards the disputed issues is supportive or dismissed," as an example, the model outputs: Entity recognition task, with a labeling system of [plaintiff's name, defendant's name, cause of action]; Element extraction task, with a labeling system of [judgment result]; Text classification task, with a labeling system of [supportive, dismissed]. The system stores these parsing results as explicitly labeled task types and their corresponding labeling systems.

[0038] S202. Obtain the attribute features of the text to be labeled, and infer the implicit labeling task type and label system. Extract attribute features for each document from the unannotated text dataset, including text type (e.g., "court documents"), domain attributes (e.g., "law"), text length (e.g., "3200 characters"), and data volume (e.g., "500 documents in total"). These attribute features can be obtained through document metadata or a lightweight classifier.

[0039] These attribute features are input into a pre-trained TextCNN classifier. During training, this classifier has learned numerous associations between text features and applicable annotation tasks. For example, for inputs with "domain = law" and "text type = judgment document," the model predicts the implicit annotation task "legal citation recognition" and its labeling system [legal citation number, legal citation content]; for inputs with "text length > 2000 characters," the model predicts the implicit task "text structure annotation."

[0040] After determining the latent annotation task type using the TextCNN classifier, a corresponding labeling system needs to be configured for that task type so that it can be uniformly included in the standardized annotation task set.

[0041] A pre-built "label mapping table" can be used, stored in a key-value pair structure, such as a JSON file, a database table, or an in-memory hash map. Each entry in the table contains two fields: a task type identifier (TaskType) and the corresponding label schema (LabelSchema). The label schema defines the set of all valid label values ​​for that task type and their semantic meaning.

[0042] Once the implicit task type is determined to be "legal citation identification", an exact lookup is performed in the label system mapping table using that string as the key. If a relational database is used, an SQL query is executed: SELECT label_schema FROM task_label_map WHERE task_type='legal citation identification'; if a memory-mapped table (such as RedisHash or a Python dictionary) is used, it is directly obtained through label_schema=mapping_table['legal citation identification'].

[0043] Upon successful lookup, the corresponding tag system dictionary is obtained. For example, for "legal provision citation recognition," the tag system {"LAW_ART":"legal provision number","LAW_CONTENT":"legal provision content"} is obtained. This tag system is packaged together with the task type to generate a complete implicitly labeled task object, which is then stored in the task list to be merged. If the lookup fails (i.e., the task type does not exist in the mapping table), an exception is logged and the administrator is prompted to update the mapping table. Simultaneously, a backup large model can be called to generate a recommended tag system for user confirmation.

[0044] These prediction results are used as implicit labeling task types and their labeling systems, and then merged with the explicit tasks obtained in S201.

[0045] S203. Generate the initial annotation task set and associate annotation objects. Explicit and implicit task types are merged and deduplicated to form an initial list of labeled task types. For each task, the system determines its labeled objects: For tasks applicable to the entire document (such as text classification and legal citation recognition), the annotation object is the entire document.

[0046] For tasks applicable to local segments (such as entity recognition and feature extraction), the annotation object is a specified segment in the document (such as the "fact section" or "reasoning section"). The system can automatically locate the segment boundaries based on the document structure (such as heading tags) or regular expression rules.

[0047] For example, the generated task set includes: "Entity Recognition (Plaintiff's Name)" with the full text labeled; "Chapter Structure Annotation" with the full text labeled; and "Legal Citation Recognition" with the full text labeled. The labeling system for each task is recorded together.

[0048] S204. Push the initial task set to the user's client for confirmation, and generate a standardized task set. The generated initial annotation task set is pushed to the user in the form of a visual interface. The interface displays each task type, tag system, and annotation objects, and provides controls for adding, deleting, and modifying. After reviewing the task, the user deems "Legal Citation Recognition" unnecessary in the current batch of documents and deletes it; at the same time, the user adds a new task, "Controversial Focus Extraction," with a tag system of [Focus Description]. The system receives the user's modification instructions, updates the initial task set, and finally generates a standardized full annotation task set for subsequent task breakdown and scheduling.

[0049] S205. Extract text features from the text to be labeled. While the user confirms the task set, the system background executes a text feature extraction process to provide semantic support for subsequent multi-model scheduling. Specifically, this includes: (1) Preprocess the text to be annotated: use regular expressions to remove garbled characters, extra spaces and HTML tags; call the langdetect library to detect the language (in this example, it is all Chinese); perform text segmentation on long text based on a sliding window (the window size is 512 tokens); use BERTTokenizer for word segmentation and tokenization.

[0050] (2) Load the pre-trained bert-base-chinese model, perform forward reasoning on each document, extract the vector at the [CLS] position as the global semantic embedding vector of the text (dimension 768), and extract the semantic feature vector corresponding to each token (dimension 768).

[0051] (3) The system will also collect text attribute features (such as extracted text type, domain, length, and data volume) and document structure features (such as number of paragraphs and number of sentences). Finally, the system will package the cleaned standardized text, global semantic embedding vector, token-level semantic features and text attribute features together and store them in a distributed shared cache (such as Redis) for subsequent task decomposition, model matching and collaborative reasoning.

[0052] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. Please refer to [the relevant documentation]. Figure 3 .

[0053] S301. Decomposing atomic operation libraries into atomic labeling tasks A predefined "atomic operation library" of basic annotation capabilities is provided, where each item corresponds to an indivisible annotation operation, such as: tokenization, part-of-speech tagging (POS), named entity recognition (NER), relation classification (RC), sentiment polarity judgment (Sentiment), text classification (TC), etc.

[0054] Each labeling task to be decomposed is judged according to the "minimum indivisibility principle": if the input of a task does not depend on the output of other labeling tasks, and its output is no longer broken down into finer-grained subtasks, then the task itself is an atomic labeling task; otherwise, it is recursively decomposed into a combination of atomic labeling tasks.

[0055] For example, the standardized task set includes the "entity relation extraction" task. This task can be broken down into two atomic annotation tasks: Atomic labeling task Named Entity Recognition (NER) outputs a list of entities. .

[0056] Atomic labeling task Relation Classification (RC), input is entity pairs Output relation type .

[0057] For example, the "text structure annotation" task can be broken down into two atomic annotation tasks: "paragraph boundary recognition" and "paragraph function classification". In this way, the system transforms all the original tasks into a single set of atomic annotation tasks. Each atom labeling task can be independently scheduled and executed in subsequent steps.

[0058] S302. Parsing Input Dependencies in Atom Labeling Tasks For each atom labeling task The system analyzes its input parameter specifications (InputSchema) and output product specifications (OutputSchema). If the task... The input parameters explicitly require the use of the task. The output result (e.g., the entity list required for entity recognition in relation classification) is then used to determine... yes Prerequisites.

[0059] Dependencies are categorized into strong sequential dependencies and weak sequential dependencies. In this embodiment, a strong sequential dependency is defined as: if task In the absence The output cannot be executed (i.e., the input parameters are completely dependent on the output). (output), then This constitutes a strong dependency. For example, a relationship classification task must wait for entity recognition to complete before it can begin; the two constitute a strong sequential dependency.

[0060] For each atom labeling task Record its direct predecessor dependency set Tasks without any prerequisites (such as text cleaning and word segmentation) satisfy the requirements. .

[0061] S303. Construct a Directed Acyclic Graph (DAG) Treating all atomic annotation tasks as nodes and dependencies as directed edges, we construct a directed graph. ,in , Since there are no circular dependencies (the absence of cycles is guaranteed by semantic constraints during task decomposition), this graph is a directed acyclic graph (DAG).

[0062] The constructed DAG can be obtained using an adjacency matrix. It indicates that its elements are defined as:

[0063] For example, for three atom labeling tasks: word segmentation ( ), entity recognition ( ), Relationship Classification ( ), the dependency relationship is , The corresponding adjacency matrix is:

[0064] This matrix explicitly describes the execution order constraints between tasks.

[0065] S304. Topological sorting and parallel execution group partitioning Perform topological sorting on the DAG to generate a linear task execution sequence, and divide the task into parallel execution groups based on the dependency hierarchy.

[0066] Define the hierarchy of each node The recurrence relation for the longest path length from any zero-in-degree node to this node is:

[0067] For example, regarding the three tasks mentioned above: , , All tasks at the same level are grouped into the same parallel execution group: Group 0 contains... (Word segmentation), Group 1 contains (Entity recognition), Group 2 contains (Relationship Classification). Since there is only one task in each group, parallelization is not possible in this example; however, in more complex scenarios, multiple independent tasks (such as word segmentation and document segmentation) may have the same hierarchy. They are thus assigned to the same parallel group and can be executed simultaneously.

[0068] The final generated task execution path includes: Parallel execution group sequence ,in .

[0069] Tasks within each group can be scheduled in parallel; tasks between groups must be executed serially in ascending index order.

[0070] For example, if there is another task, "sentiment analysis" ( It does not depend on any task. Then the 0th parallel group is The system will simultaneously initiate word segmentation and sentiment analysis tasks, thereby significantly shortening the overall processing time.

[0071] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0072] S401. Candidate Model Selection and Constraint Modeling For the current atomic annotation task (e.g., "named entity recognition"), all models whose annotation task type includes "named entity recognition" are selected from the pre-built model profile library to form an initial model set. The model profile library records various metrics for each model, including annotation accuracy. Inference delay Single sample computing power consumption Domain adaptability wait.

[0073] Hard and soft constraints are generated using user-submitted performance constraint parameters. Assuming the user selected "Fast Mode," the resulting performance constraint parameter is: Maximum Inference Latency. Maximum video memory usage Minimum annotation precision Hard constraints are defined as follows:

[0074] i.e., model All three inequalities must be satisfied simultaneously, otherwise Those that fail to meet the requirements are directly eliminated. Soft constraints are used to quantify how well the model conforms to user preferences. If the user preference is "low latency," then the soft penalty function is defined as:

[0075] in This is the penalty coefficient. Models with higher latency receive larger soft penalty values, thus putting them at a disadvantage in subsequent scoring.

[0076] After applying hard constraints, the initial model set The candidate model set consists of all models that meet the hard conditions. .

[0077] S402.TOPSIS Multi-Attribute Decision-Making Comprehensive Scoring The candidate model set was evaluated using the following metrics: annotation accuracy (the higher the better), inference latency (the lower the better), computational power consumption (the lower the better), and domain fit (the higher the better). Perform TOPSIS scoring.

[0078] Constructing a decision matrix ,in Indicates the model index. This represents four indicators. Vector normalization is performed on the matrix:

[0079] Determine the ideal optimal solution based on indicator attributes. worst-case scenario and ideal solution :

[0080] Among them, accuracy and domain adaptability are benefit-oriented indicators (the higher the better); latency and computing power consumption are cost-oriented indicators (the lower the better).

[0081] Calculate the Euclidean distance from each model to the positive and negative ideal solutions:

[0082] Based on this, the soft constraint penalty term is directly incorporated into the distance calculation. Assume the soft penalty value is... The corrected formula for calculating relative closeness is:

[0083] Select after correction The largest model is used as the recommendation model in the TOPSIS stage. When multiple models... When they are very close, keep the first one. The model is provided for further optimization through reinforcement learning.

[0084] S403. Deep Q-Network (DQN) Reinforcement Learning Dynamic Scheduling The DQN algorithm is adopted, with the optimization goals of improving annotation accuracy, reducing overall inference time, and increasing resource utilization, to dynamically adjust the model selection strategy. State space. This includes the current system load (such as GPU utilization and request queue length), the text features of the tasks to be processed, and the remaining performance margin. Action Space To select a specific model instance from the candidate models.

[0085] reward function Designed as follows:

[0086] in: This represents the improvement in average accuracy compared to the previous round after selecting the model; This represents the increase in inference latency; This represents the increase in video memory usage. As a penalty term, when the selected model violates any hard or soft constraints (e.g., delay exceeding limits), Otherwise, it is 0; These are positive weighting coefficients that are dynamically adjusted based on user patterns, such as in fast mode. and Choose a larger value to emphasize speed and compliance.

[0087] The Q-function update follows the standard Bellman equation:

[0088] in For learning rate, This serves as a discount factor. The system is pre-trained offline using historical scheduling data and continuously fine-tuned during online runtime. After multiple iterations, the DQN agent learns to automatically select the optimal model that satisfies performance constraints while balancing accuracy and efficiency in high-concurrency scenarios.

[0089] Finally, output the optimal matching model corresponding to the k-th atom labeling task. K represents any atom labeling task.

[0090] S404. Scheduling Execution Sequence Generation After obtaining the optimal model for all atomic annotation tasks, the models are arranged into a scheduled execution sequence according to their dependencies based on the task execution paths (DAG and parallel execution groups). For models belonging to the same parallel execution group... For multiple atomic annotation tasks, execution instructions are issued concurrently to the corresponding model instances; for cross-group tasks, the next group is started only after all tasks in the previous group have been completed.

[0091] For example, parallel execution groups It simultaneously calls word segmentation models (such as HanLP) and document segmentation models (such as rule-based segmenters). The system is waiting. Once all steps are complete, the entity recognition model (such as BERT-CRF) is invoked. The model scheduling execution plan obtained through this orchestration is then submitted to the multi-model collaborative execution layer for actual inference.

[0092] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0093] In this embodiment, after multi-model collaborative inference is completed, the original annotation results output by each atomic annotation model are obtained. Before fusing these results, the system executes the following... Figure 4 The triple consistency verification process shown is implemented as follows.

[0094] S501. First layer of verification: Confidence-weighted verification Suppose that for a given labeled object (e.g., a text mentioning "Zhang San"), multiple labeling models output different results. We retrieve the domain adaptation weights of each model from the model profile library for the task type (e.g., "named entity recognition") of the labeled object. And obtain the confidence level of the current result from the model's output. For the first Calculate the weighted score of the output of each model:

[0095] in , All candidate results are weighted and scored. Sort from highest to lowest to obtain the sequence. .set up The highest score is achieved if there exists a unique highest score that satisfies:

[0096] in For a preset threshold of significant difference (e.g.) If the result is the highest score, then the result corresponding to that score is directly determined as the one that passes the first level of validation and is used as the final output, without needing to proceed to subsequent validations. Otherwise, the top scores are retained based on weighted average. Results (e.g.) ) or all scores less than the highest score The results constitute the set of results to be verified. .

[0097] S502. Second verification: Rule consistency verification Maintain a predefined annotation rule base, which contains multiple semantic and logical consistency rules. For example: Relation extraction rule: Relation triples The main body in and object It must also exist in the entity recognition result list of the current text segment.

[0098] Sentiment analysis rule: If the sentiment polarity of a certain opinion evaluation word is "positive", then the sentiment label associated with that word cannot be "negative".

[0099] Event extraction rules: The type of event element (such as "time", "location", "person") must match the element pattern predefined for that event type.

[0100] For the result set to be verified Each result in Each result is checked individually to see if it violates any of the above rules. If a violation is found, the result is filtered out. After rule validation, a valid result set is obtained. .

[0101] like If the result is not empty, then the result with the highest weighted score is selected as the result that passes the second verification, denoted as . And use it as the final output.

[0102] like If all candidate results are filtered by the rules, then the third layer of verification is triggered.

[0103] S503. Third-level verification: Large model arbitration verification The third level of verification will be initiated when one of the following two conditions is met: The weighted score difference among the multiple results retained after the first verification is less than or equal to... This makes it impossible to uniquely determine the optimal result based on confidence level; After the second round of validation, all results were filtered by the rules, and no valid results were found.

[0104] At this point, constructing an arbitration request specifically includes: Original text fragment (i.e., the text area that caused the conflict); A collection of all conflicting task annotation results (It can be either all the results to be verified after the first verification or the original conflict results before the second verification.) Relevant annotation rules (For example, a description of the rule being violated and a positive example).

[0105] This information is then combined into a prompt according to a preset template, for example: As a text annotation arbitration expert, please make a final judgment on the following conflict results: Original text: {original fragment}; Candidate result 1: {result 1}, confidence level {conf1}; Candidate result 2: {result 2}, confidence level {conf2}; Annotation rule: {rule description}; Please output the annotation result you believe to be correct and give a brief reason.

[0106] The prompt is sent to a pre-trained large language model arbitrator (e.g., Tongyi Qianwen or LLaMA-3-70B). Based on its own knowledge and the provided rules, the arbitrator outputs the final arbitration result. .

[0107] S504. Integration and Output of Verification Results Based on the results of the three-fold verification, the verified task annotation result for each atomic annotation task is determined: if the first-fold verification has directly determined a unique result, then that result is adopted; otherwise, if the second-fold verification produces a valid result, then that result is adopted; otherwise, the arbitration result of the large model from the third-fold verification is adopted.

[0108] In terms of form, the final output It can be represented as:

[0109] This final result will serve as a reliable annotation for the labeled object, and will be used for subsequent multi-dimensional result fusion and structured output.

[0110] After completing triple consistency checks on the multi-model annotation results, the system obtains verified annotation results corresponding to multiple atomic tasks. These results may come from different models and have different data granularities and representations. To generate user-oriented, full-dimensional structured annotation output, the system performs the following fusion and delivery steps.

[0111] S505. Spatial Alignment Since different annotation models may output annotation objects in different ways in the original text (for example, entity recognition models output character start and end indices, while syntactic analysis models output token-level offsets), spatial alignment is required to ensure that all annotations point to the same region in the text.

[0112] Specifically, the implementation involves coordinate normalization based on a unified text segment identifier (segment ID, sentence ID) and token offset. Each annotation result contains three fields: "unique text identifier," "starting token position," and "ending token position." All validated annotation results are iterated through, and the positional information of each annotation is mapped to the same standardized text coordinate system.

[0113] For example, when the entity recognition model outputs "Zhang San" at character position [5,7], and the sentiment analysis model outputs sentiment tags bound to the same "Zhang San" entity, the character position is converted into a token index through the mapping table between characters and tokens (assuming characters 5-7 correspond to token index 2), thereby enabling the annotations of the two different models to be spatially bound to the same object.

[0114] S506. Semantic Association Binding After spatial alignment, the semantic relationships between the annotation results are further constructed. This is a crucial step in forming a structured annotation map. Based on the annotation type and predefined association rules, links between annotations of different dimensions are established: Entity-Relation Association: For the relation triples (head entity, relation type, tail entity) output by the relation extraction task, the unique identifiers of the head entity and tail entity are searched in the entity identification result list, and the relation label is linked to these two entity labels. If an entity is in the entity identification result, a "relation as head entity" field is added to the attribute list of that entity, pointing to the corresponding relation label.

[0115] Entity-Sentence / Opinion Association: For sentiment analysis or opinion mining tasks, the output includes "evaluation object" (e.g., "screen"), "evaluation word" (e.g., "very clear"), and "sentiment polarity" (e.g., "positive"). The text fragment of the evaluation object is matched with the entity in the entity recognition result. If the match is successful, the sentiment label is bound to the corresponding entity, generating an "entity-sentiment" attribute pair.

[0116] Association between event elements and entities: Each event output by the event extraction task contains several elements (such as "time", "location", and "participants"). The value of each element (such as "January 1, 2024") is matched with the time expression in the entity recognition result, and the element is labeled and linked to the corresponding entity node.

[0117] The association between text structure and paragraphs / sentences: The paragraph function tags (such as "factual statement", "legal basis") and sentence logical relationships (such as "cause and effect", "transition") output by the text annotation model are directly bound to the corresponding areas of the text through location information (paragraph index, sentence index).

[0118] Through the above connections, a knowledge graph structure was constructed, with text fragments, entities, events, sentiments, and relationships as nodes, and relationships such as "containment," "pointing to," and "evaluation" as edges. This graph serves as an intermediate representation for full-dimensional structured annotation.

[0119] In some embodiments, the multi-model collaborative text annotation system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the multi-model collaborative text annotation system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) A text annotation function based on multi-model collaboration.

[0120] In this embodiment, the text annotation system based on multi-model collaboration can be divided into multiple functional modules according to the functions it performs, such as... Figure 5 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0121] The parameter acquisition module is used to acquire the text to be labeled, labeling requirements, and performance constraint parameters submitted by the user. The task generation module is used to parse the text to be labeled and the labeling requirements, generate a standardized labeling task set, and extract the text features of the text to be labeled. The task decomposition module is used to decompose the annotation tasks in the standardized annotation task set into multiple atomic annotation tasks, and construct task execution paths according to the dependencies between the atomic annotation tasks. The model scheduling module is used to match the optimal annotation model for each atomic annotation task based on the text features, the performance constraint parameters and the pre-built model profile library, using a multi-attribute decision algorithm, and to generate a model scheduling execution plan based on the task execution path and the matched annotation model. The result processing module is used to call multiple annotation models to perform collaborative inference according to the model scheduling execution plan, obtain the task annotation results output by each annotation model, and perform fusion processing on the multiple task annotation results to generate and output the annotation results.

[0122] Figure 6The text annotation method based on multi-model collaboration provided in this application embodiment can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device. The device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Specifically, the device 600 may include: a processor 610, a memory 620, and a communication unit 630. These components communicate through one or more buses. Those skilled in the art will understand that the server structure shown in the figures does not constitute a limitation on the invention; it can be a bus topology or a star topology, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0123] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0124] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0125] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0126] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0127] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0129] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A text annotation method based on multi-model collaboration, characterized in that, include: Obtain the text to be annotated, annotation requirements, and performance constraint parameters submitted by the user; The text to be labeled and the labeling requirements are parsed to generate a standardized labeling task set, and the text features of the text to be labeled are extracted. The annotation tasks in the standardized annotation task set are decomposed into multiple atomic annotation tasks, and task execution paths are constructed based on the dependencies between the atomic annotation tasks. Based on the text features, the performance constraint parameters, and the pre-built model profile library, a multi-attribute decision algorithm is used to match the optimal annotation model for each atomic annotation task, and a model scheduling execution plan is generated based on the task execution path and the matched annotation model. According to the model scheduling execution plan, multiple annotation models are invoked to perform collaborative inference, the task annotation results output by each annotation model are obtained, and the multiple task annotation results are fused to generate and output the annotation results.

2. The method according to claim 1, characterized in that, Obtain the user-submitted text to be annotated, annotation requirements, and performance constraint parameters, including: Receive user-uploaded datasets of text to be labeled; Receive annotation requests from users in natural language format; The annotation mode selected by the user is received and mapped to specific performance constraint parameters. The performance constraint parameters include at least one or more of the following: annotation accuracy priority, inference latency requirements, deployment environment limitations, computing power resource thresholds, and data compliance requirements.

3. The method according to claim 1, characterized in that, The text to be annotated and the annotation requirements are parsed to generate a standardized annotation task set, including: Semantic parsing is performed on the annotation requirements to extract one or more explicit annotation task types and the corresponding tag system for each type; Obtain the attribute features of the text to be labeled, and infer one or more implicit labeling task types and the corresponding label system for each type based on the attribute features; The explicit annotation task type and the implicit annotation task type are merged to generate an initial annotation task set for the text to be annotated, wherein each annotation task is associated with the text to be annotated or a partial fragment thereof as an annotation object; The initial annotation task set is pushed to the user for confirmation, and a standardized annotation task set is generated based on the user feedback.

4. The method according to claim 3, characterized in that, Obtain the attribute features of the text to be labeled, and infer one or more implicit labeling task types and the corresponding label system for each type based on the attribute features, including: Extract the attribute features of the text to be labeled, wherein the attribute features include at least domain attributes and text length; The attribute features are encoded into feature vectors and input into a pre-trained TextCNN classifier, which includes convolutional layers, pooling layers, and fully connected layers. The TextCNN classifier outputs the probability distribution of each candidate latent labeling task, and determines the candidate labeling task type with a probability higher than a preset threshold as the latent labeling task type. Based on the determined implicit annotation task type, the label system corresponding to the task type is retrieved from the pre-built label system mapping table and used as the label system for the implicit annotation task.

5. The method according to claim 1, characterized in that, The annotation tasks in the standardized annotation task set are decomposed into multiple atomic annotation tasks, and task execution paths are constructed based on the dependencies between the atomic annotation tasks, including: Based on a predefined atomic operation library, each annotation task is decomposed into indivisible atomic annotation tasks according to the principle of minimum indivisibility; The input dependencies of each atomic labeling task are analyzed. The input dependencies include: if the input parameters of one atomic labeling task depend on the output of another atomic labeling task, then it is determined that there is a strong serial dependency between the two. A directed acyclic graph is constructed based on the aforementioned dependencies, where nodes represent atomic labeling tasks and directed edges represent dependencies between tasks. The directed acyclic graph is topologically sorted to generate acyclic task execution sequences, and parallel execution groups are divided according to dependency levels, wherein atomic labeling tasks without prerequisite dependencies are assigned to the same parallel execution group.

6. The method according to claim 1, characterized in that, Based on the text features, performance constraint parameters, and a pre-built model profile library, a multi-attribute decision algorithm is used to match the optimal annotation model for each atomic annotation task. A model scheduling execution plan is then generated based on the task execution path and the matched annotation model, including: For each atom annotation task, an initial set of models suitable for the task is selected from a pre-built model profile library; Hard constraints and soft constraints are generated using the performance constraint parameters, wherein the hard constraints are used to define the mandatory indicator boundaries that the model must meet, and the soft constraints are used to measure the degree to which the model meets user preference indicators. By applying the hard constraints, any model in the initial model set that does not meet the mandatory index boundary is eliminated, and the remaining models constitute the candidate model set. Using annotation accuracy, inference latency, computing power consumption, and domain adaptability as evaluation indicators, the TOPSIS multi-attribute decision algorithm is used to comprehensively score each model in the candidate model set. The comprehensive score is obtained by calculating the distance between each model and the ideal optimal model and the ideal worst model. The closer the model is to the ideal optimal model and the farther the model is from the ideal worst model, the higher the comprehensive score. In addition, when calculating the distance, a score penalty is imposed on the model that deviates from the user preference index according to the soft constraint condition. Combining the deep Q-network reinforcement learning algorithm, with the optimization objectives of improving annotation accuracy, reducing overall inference time, and improving resource utilization, and incorporating behaviors that violate the hard or soft constraints as negative reward terms into the reward function, the model selection strategy is dynamically adjusted through iterative learning to output the optimal matching model for each atomic annotation task; Based on the dependencies of each atomic annotation task in the task execution path, the matched models are arranged into a scheduling execution sequence according to the dependencies, and a model scheduling execution plan is generated.

7. The method according to claim 1, characterized in that, Before fusing the annotation results of multiple tasks, the method further includes: For cases where multiple models output different annotation results for the same labeled object, the domain adaptation weight of each model in the task type to which the labeled object belongs and the confidence of the current output result are obtained. The domain adaptation weight and the confidence are weighted and calculated. The candidate results are sorted according to the weighted score, and one or more results with the highest weighted score are retained as the set of results to be verified. If the result with the highest weighted score is unique and its score is significantly higher than other results, then the result is directly determined as the result that passes the first verification. For each result in the set of results to be verified, a validity check is performed based on a predefined annotation rule base, and results that violate any rule are filtered out. The rules include at least the following: the subject and object in the relation triple must both exist in the entity list; the sentiment annotation result must be consistent with the sentiment tendency of the corresponding opinion evaluation word; and the type of the event element must match the predefined element type of the event. If at least one valid result still exists after rule verification, the result with the highest weighted score is selected as the result that passes the second verification. When the score difference between the highest weighted scores after the second verification is less than a preset threshold, or when all results are filtered by rules after the second verification, resulting in no valid results, the corresponding original text fragment, all conflicting task annotation results, and related annotation rules are input into the pre-trained large language model arbitrator, and the large language model outputs the final arbitration result. Based on the results of the first verification, the second verification, and the arbitration result determined after the third verification, output the verified task labeling results corresponding to each atomic labeling task.

8. A text annotation system based on multi-model collaboration, characterized in that, include: The parameter acquisition module is used to acquire the text to be labeled, labeling requirements, and performance constraint parameters submitted by the user. The task generation module is used to parse the text to be labeled and the labeling requirements, generate a standardized labeling task set, and extract the text features of the text to be labeled. The task decomposition module is used to decompose the annotation tasks in the standardized annotation task set into multiple atomic annotation tasks, and construct task execution paths according to the dependencies between the atomic annotation tasks. The model scheduling module is used to match the optimal annotation model for each atomic annotation task based on the text features, the performance constraint parameters and the pre-built model profile library, using a multi-attribute decision algorithm, and to generate a model scheduling execution plan based on the task execution path and the matched annotation model. The result processing module is used to call multiple annotation models to perform collaborative inference according to the model scheduling execution plan, obtain the task annotation results output by each annotation model, and perform fusion processing on the multiple task annotation results to generate and output the annotation results.

9. A text annotation device based on multi-model collaboration, characterized in that, include: Memory is used to store text annotation programs based on multi-model collaboration; A processor, configured to implement the steps of the multi-model collaborative text annotation method as described in any one of claims 1-7 when executing the multi-model collaborative text annotation program.

10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a text annotation program based on multi-model collaboration, which, when executed by a processor, implements the steps of the text annotation method based on multi-model collaboration as described in any one of claims 1-7.

Images