Man-machine collaborative text data coding method

Abstract

The invention provides a man-machine collaborative text data coding method. The method specifically comprises the following steps of S1, formulating a code book; s2, constructing a coding model; and S3, verifying the coding saturation. According to the method, the manual coding workload is effectively reduced, and the coding sample representativeness is improved. Meanwhile, a multi-agent data enhancement method is provided, and the key problems of label imbalance, sample finiteness, sample singleness and the like are solved. Besides, the dynamic integrated attention voting system provided based on the simulation thought of the real coding and discussion process focuses on complementarity between models, not only is the training deviation of a single high-performance model weakened, but also the actual effect of the model can be further improved. Finally, the invention provides a complete verification standard of the coding saturation, and the problem breakthrough of quality interpretation-quantitative verification is realized.

Description

Technical Field

[0001] The present invention relates to the field of data annotation technology, specifically a human-computer collaborative text data encoding method, which includes three parts: first, formulating a codebook; second, building a coding model; and third, verifying coding saturation. Background Art

[0002] The booming development of digital technology has created a more urgent demand for diverse, high-quality data. With the continuous rise in overall computing power and the rapid advancement of transmission technologies and engines, the storage and transmission costs of text data have significantly decreased, leading to the emergence of large-scale text corpora such as annual reports and online reviews. Text information has a high information density and contains rich insights, emotions, and psychological intentions. Its deep mining places even more stringent demands on data processing and algorithm modeling capabilities.

[0003] Coding is a fundamental step in extracting text information. It aims to continuously analyze, summarize, and label text fragments such as sentences and paragraphs in in-depth interview materials, abstractly representing the people, events, concepts, and categories through concise phrases or keywords. Commonly used coding strategies include manual coding, automatic coding based on large-scale language models, and machine coding using supervised learning.

[0004] Manual coding involves manually labeling and annotating text corpora. Its advantage lies in the ability of humans to discern pragmatic cues and contextual details, allowing them to accurately discern the underlying semantics of a language. However, this method is costly, has limited repeatability, and is difficult to scale for massive corpora. Collaborative efforts can also lead to a loss of consistency due to differences in subjective standards, and these issues become increasingly prominent as text complexity increases. Furthermore, manual annotation lacks a quantitative assessment of coding saturation, limiting the verifiability and comparability of the results.

[0005] Automatic encoding based on large language models (such as DeepSeek, ChatGPT, etc.) relies on the excellent semantic inference capabilities of pre-trained models. With only a small number of examples or even just task descriptions, they can quickly migrate and complete text annotation in new domains or custom labeling systems. Despite this, the attention distribution of such models has a certain degree of uncertainty, which can easily lead to overfitting or "over-recognition". Especially when dealing with long texts with complex structures, the generalization ability and robustness shown are still insufficient. If you want to alleviate the above limitations, it is often necessary to deeply fine-tune the model and reshape the parameters, which has a high technical threshold and implementation difficulty.

[0006] Supervised learning relies on manually standardized training and test sets, iteratively optimizing model parameters to achieve ideal performance on a validation set before scaling to massive sample sizes. This paradigm combines excellent scalability with computational efficiency, along with a transparent methodology and reproducible processes, providing a viable path for a closed loop of "qualitative interpretation and quantitative verification." However, this approach is highly dependent on the representativeness, scale, and quality of labeled data; even with a single model that achieves excellent predictive performance, it is difficult to completely eliminate training bias. Therefore, the model's universality and credibility remain questionable in large-scale labeling scenarios.

[0007] To overcome the limitations of existing technologies, the present invention proposes a human-machine collaborative text encoding scheme based on the supervised learning encoding framework: First, the present invention constructs a "coding book" to systematically improve the representativeness and coverage of the training set and test set. Among them, the text generation and data enhancement mechanism based on multi-agents further expands the data scale and enhances the generalization ability of the model; secondly, a dynamic integrated attention voting system is constructed to achieve bagging-style integrated learning through simulated voting, thereby reducing the training bias of a single high-performance model; finally, a quantitative verification method for encoding saturation is proposed based on scene heterogeneity. This method improves the consistency, accuracy and verifiability of text encoding as a whole, and provides a new technical path for high-quality annotation of large-scale and complex corpora. Summary of the Invention

[0008] To address the challenges inherent in existing technologies, this paper proposes a collaborative human-machine text data encoding method. By creating a codebook, this method significantly improves the representativeness and scale of training and test data, enhancing the model's transferability and generalization capabilities. Furthermore, a dynamic integrated attention voting system is constructed to effectively mitigate training biases in individual models and improve overall prediction robustness. Furthermore, a quantitative verification method based on scene heterogeneity in encoding saturation provides verifiable control over output quality, further ensuring the consistency and reproducibility of annotation results.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A human-computer collaborative text data encoding method comprises the following steps:

[0011] Step S1: Create a code book:

[0012] This step aims to select the most representative text files from the text data for manual coding, reducing the manual coding workload while ensuring the representativeness of the codes. Simultaneously, the multi-agent data augmentation system proposed in this invention is applied to further increase the data scale and the generalization ability of the model. To this end, the codebook preparation step proposed in this invention includes the following three stages:

[0013] (1) Encoded text screening stage: In this stage, the potential topic structure of the corpus is first revealed with the help of clustering algorithms, and the probability distribution of each text on each topic is calculated.

[0014] (2) Manual coding stage: Based on the probability matrix, the most representative samples of the topic are selected to enter the coding process. Multiple annotators annotate each sentence in a back-to-back manner, and hold centralized discussions according to the preset period to resolve differences and unify the coding results.

[0015] (3) Multi-agent data enhancement stage: First, a variety of large-scale language models are used to extract and refine each label and its semantic connotation to form a standardized and operational label definition; then a multi-agent collaborative mechanism is introduced to carry out text generation and enhancement, and the sample size of each label system is balanced.

[0016] Step S2: Constructing a coding model:

[0017] This step aims to apply a dynamic integrated attention voting system to simulate the back-to-back coding process and integrated discussion process of real-life coders. In this step, each basic model is regarded as an independent coder, and the dynamic integrated attention voting system is used to simulate the discussion and voting mechanism, thereby alleviating the objective bias that may exist under a single model system and enhancing the adaptability and generalization ability of the coding model. To this end, the steps of constructing a coding model based on a dynamic integrated attention voting system proposed in the present invention include the following two stages:

[0018] (1) Model training stage: Based on the codebook formed in step S1, each classifier model is trained.

[0019] (2) Dynamic integrated attention voting stage: After each classifier model completes training and builds a candidate classifier pool, the model's "synergy ability" standard is introduced to calculate the Shapley value of each classifier. For a specific sample, classifiers with positive Shapley values ​​are selected to participate in the voting. Furthermore, a dynamic weighting mechanism based on the attention mechanism is constructed, which can dynamically adjust the voting weights of different basic models according to their confidence scores. Specifically, the attention mechanism is used to determine which basic models have higher importance in the prediction of a specific sample, thereby simulating a real multi-party discussion and voting scenario.

[0020] Step S3: Verify coding saturation

[0021] This step aims to encode the remaining text while identifying and training the missing labels and samples, thereby quantifying the theoretical saturation of the text encoding and ultimately achieving a logical closed loop of "qualitative analysis-quantitative verification". To this end, the text data encoding saturation verification step proposed in this invention includes the following two stages:

[0022] (1) Text encoding and judgment phase: Based on the trained encoding model, the remaining text is predicted. Samples that cannot be effectively classified are marked for subsequent analysis. For samples that cannot be classified, the reasons for their classification failure are deeply analyzed. Labels and related samples that are not included in the existing encoding system are re-incorporated into the encoding book after the multi-agent data enhancement phase to enrich the model's label library.

[0023] (2) Model iteration phase: The newly included codes and their samples are re-learned using the trained coding model to optimize the model’s performance and adaptability. The re-learned coding model is then used to predict the remaining samples. Even if there are still samples that cannot be effectively classified at this stage, if the reason for their classification failure does not include the appearance of new labels, and if there is no significant deviation in the coding results of the successfully identified samples, the text coding can be considered to have reached saturation, and the coding process is terminated.

[0024] Compared with the existing methods, the present invention has the following advantages:

[0025] 1. Through the step of formulating a coding sheet, the present invention, on the one hand, reveals the potential topic structure of text data through a clustering algorithm, reduces the workload of manual coding while ensuring the representativeness of the coding, thereby improving efficiency; on the other hand, constructs a data enhancement mechanism under the multi-agent collaborative technology architecture, expands the data scale, and at the same time improves the generalization ability of the model.

[0026] 2. The present invention selects classifiers with positive Shapley values ​​to participate in voting based on the calculated Shapley value of the classifier, maximizing the overall model performance. At the same time, it dynamically adjusts the voting weight according to the confidence score of the basic model, simulating real multi-person discussion and voting scenarios, and increasing the adaptability and accuracy of the model.

[0027] 3. This invention utilizes the high scalability and computational efficiency of supervised learning, clarifies the saturation standard of text encoding, provides a clear and reproducible path for the "qualitative interpretation-quantitative verification" closed loop, and enhances the credibility of text data encoding results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart for text data encoding for human-machine collaboration;

[0029] Figure 2 Multi-agent data augmentation flowchart. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0031] The present invention provides a human-computer collaborative text data encoding method, such as Figure 1 As shown, the method includes the following steps.

[0032] Step S1: Create a code book:

[0033] In the encoding text screening stage, the text data should first be segmented into natural segments and subjected to text preprocessing such as removing stop words. Furthermore, each document i is represented as a vector containing unique word frequencies based on the bag-of-words method, as shown in formula (1):

[0034] W i =(W i1 ,W i2 ,W i3 ,…,W iM ) (1)

[0035] The Euclidean distance is used to measure the similarity between documents, as shown in formula (2) and formula (3):

[0036]

[0037] Among them, c k represents the cluster center of the kth cluster, which is calculated by the average value of the features of all documents in the cluster on the corresponding dimension. km is the cluster center c k The value on the mth feature, that is, the coordinate of the cluster center of the kth cluster on the mth feature dimension. im is the value of document i on the mth feature, c k is the set of all documents in cluster k, n k is the number of documents in cluster k.

[0038] According to the objective function f(c,C,W), the distance between each document and its corresponding cluster center is minimized, as shown in formula (4):

[0039]

[0040] Among them, I(C i =k) ​​is an indicator function. When a document is assigned to cluster k, its value is 1, otherwise it is 0. The K-means optimization algorithm is used to iteratively optimize the clustering results. First, each document is assigned to the nearest cluster center as shown in formula (5), and the cluster center is updated according to the newly assigned cluster using formula (6).

[0041]

[0042] Calculate the probability of each document in different clusters and define the probability that document i belongs to cluster k as:

[0043]

[0044] The specific calculation of each component is as follows, including: conditional probability, prior probability and marginal probability.

[0045]

[0046] Where k in Equation (8) is a proportional constant. Finally, the probability distribution matrix P(C|W) is formed by combining the document and its probability under each cluster.

[0047] During the manual coding phase, documents were ranked according to their probability under different topics, and the most representative documents within each topic were selected for manual coding. Specifically, the top N documents with the highest probability distribution under each topic were selected for coding (N ≥ 10). To facilitate subsequent model training, the coding process adopted a sentence-by-sentence coding approach, with two or more coders conducting back-to-back open coding. Coders discussed the coding process at a fixed coding step, achieving axial coding to clarify the labeling system for each data sample.

[0048] In the multi-agent data enhancement phase, first Figure 2 As shown, large language models such as Deepseek and ChatGPT are used to read the encoding results and their corresponding text data, summarizing the concepts and meanings of each encoding. Coders review, modify, and discuss the generated content to ensure accuracy and consistency. The iterative results are then evaluated by the relevant large models. When all large models fail to provide substantive optimization suggestions, the label definition is considered saturated.

[0049] Subsequently, text generation is performed by introducing a multi-agent technology framework to achieve a balance in the amount of text under different labels. At the same time, in order to alleviate the bias that may occur in large models, each agent needs to use an API interface independently and restrict Temperature. The multi-agent technology framework proposed in the present invention includes three roles: text generator, text analyst and natural language training expert. Among them, the role positioning of text generator and text analyst can be adjusted accordingly according to the different text types. In order to avoid interfering with the text content under other label systems, this stage adopts a one-by-one generation strategy and uses a single label system for text generation.

[0050] During the text generation process, the text generator needs to conduct a comprehensive review and analysis of the text under the tag system, focusing on its linguistic style and rewriting the text according to the tag definition. The goal of text generation is to generate a rewritten text with a limited number of words, or to effectively combine the two methods.

[0051] The text analyst's responsibility is to rigorously judge and analyze the generated sentences, taking into account the label definitions, to confirm that the generated content remains within the label's scope. If the generated content deviates from the label requirements, the analyst will provide specific, actionable revision suggestions, which the text generator will then make accordingly. This process is repeated three times. If the generated text no longer deviates from the label's scope within these three revisions, it is considered passed and submitted to natural language training experts for further evaluation. If deviations persist, the text is deleted and regenerated.

[0052] During the evaluation process, natural language training experts will combine the original text and label definitions to determine whether the generated sentences introduce significant training noise. If noise is found, the experts will provide specific and actionable revision suggestions, and the text generator will make corresponding corrections. This process will also be repeated three times. If the generated text no longer deviates from the label range after three revisions, it will be considered passed; otherwise, the text will be deleted and regenerated.

[0053] Furthermore, the present invention proposes a data enhancer structure and further designs a data processing function. This function first injects the original text and the generated text into the processing flow to count the amount of data under each label and compare it with the preset target number to determine the number of samples required for enhancement for each label.

[0054]

[0055] Among them, S l represents the text input set; L is the label set; T l represents the number of preset targets for each label l∈L; C l Indicates the current number of samples for each label; N l Indicates the number of samples that need to be enhanced for each label. l When is > 0, the data enhancer is called to perform data enhancement on the text input set. The data enhancement logic constructed in this invention includes translation enhancement, synonym replacement, homophone replacement, equivalent character replacement, and character position swapping. The translation enhancement logic performs a translation process from Chinese to a random foreign language, then to English, and then back to Chinese.

[0056]

[0057] S aug=Augment(S l ,M) (17)

[0058] S final =S l ∪S aug (18)

[0059] According to formula (14)-formula (18), for each s i ∈S l , randomly select 2 to 4 enhancement methods for a certain sample and use them in combination, and set the perturbation coefficient of each enhancement method in the range of 0.3 to 0.5 to ensure the diversity of the generated text, thereby effectively improving the generalization ability and robustness of the model and forming the random enhancement function result Augment(s,M). l After repeating the above process, a data enhancement set S is formed aug Finally, S final S l and S aug The combined data sets are used as codebooks.

[0060] Step S2: Constructing a coding model:

[0061] During the model training phase, a pool of four or more classifiers needs to be constructed, as shown in formula (19):

[0062] C={c1,c2,…c k},k≥4 (19)

[0063] Among them, C is the classifier pool, which contains all available classifiers. k Represents different classifiers. The reason for k ≥ 4 is that in order to give full play to the mechanism of the ensemble model, it is necessary to broaden the diversity and richness of the classifiers, thereby forming complementarity between different models. Each classifier needs to be trained and adjusted separately.

[0064] The dynamic ensemble screening mechanism relies on the collaborative capability contribution value of each classifier and further calculates the Shapley value. After obtaining the Shapley value of each classifier, the classifiers with positive Shapley values ​​are selected to construct the final classifier set. The collaborative capability contribution value of each classifier is shown in Equation (20)-Equation (21):

[0065]

[0066] in, Represents classifier c i , considering c i For a classifier combination G i Specifically, it means c iCombined with classifier G i The performance improvement brought about by their combined effect. Represents classifier c i The overall collaborative ability of the classifier is considered, which comprehensively considers the influence of the combination G set with all classifiers; n represents the number of effective combinations; w s Representative in collaboration with group G c i The weight of V(S i ) is the current classifier combination G i The utility function of , usually represents the accuracy of classification.

[0067] Furthermore, after integrating the collaborative ability contribution values ​​of all candidate classifiers, the Shapley value is calculated according to formula (22).

[0068]

[0069] Represents the candidate classifier c i The Shapley value represents the classifier's i Importance in decision making; Π(C) represents the set of all possible permutations and combinations in the classifier pool; Represents the arrangement of π t In the classifier c i The set of all previous classifiers. Select The classifier constructs the final classifier set. If there is no Then mark it as recognition failure.

[0070] Furthermore, based on the traditional idea of ​​weighted voting based on Shapley value, this paper introduces an attention mechanism to further adjust the prediction of each basic model through the attention weight learned by the model, and constructs a voting function, as shown in formula (23):

[0071]

[0072] Among them, ρ(x j ) represents the final predicted category label, which means that the class with the highest vote is selected as the sample x j The prediction results of h·(y i |s j ) is for the input sample x j The voting function combines the attention mechanism and the Shapley weighted voting results, as shown in Equations (24)-(25):

[0073]

[0074] Among them, I(h l (yi |s j ) is the indicator function if the model c l Prediction sample s j For category y i , then I=1, otherwise I=0; For model c l The Shapley weight of model c l The relative contribution of the model is based on its performance and marginal contribution in different sample combinations; Attention l (s j ) is the model c l The attention weight for sample s j , model c l The trust or importance of is calculated as shown in formula (25):

[0075] Attention l (x j )=softmax(W q ·h l (x j ))(25)

[0076] Among them, W q is the weight matrix obtained from the training results, h l It is model c l For input x j Representation.

[0077] This paper proposes a dynamic integrated attention voting system based on Shapley values ​​and the attention mechanism. This system leverages the strengths of existing models by dynamically evaluating the contribution and trustworthiness of each model while flexibly adapting to the importance of different inputs. This approach not only improves overall model performance but also enhances the model's decision-making capabilities.

[0078] Step S3: Verify coding saturation

[0079] In the text encoding and judgment stage, the trained encoding model is first applied to encode the remaining text data. It should be noted that considering the semantic complexity of the text data, the encoding conditions of the model need to be relaxed during the model encoding process, that is, one text data can return multiple encoding results. j ,none It is marked as recognition failure. The reason is that there is no This means that no classifier can play a positive role in predicting the content of the text.

[0080] Furthermore, for texts that fail to be recognized, coders are introduced to conduct analysis and judgment. The present invention summarizes the reasons for recognition failure into the following three categories: the first problem is irrelevant topics, that is, the text content is not relevant to the target topic; the second problem is that the text is overly complex, which means that the text structure and semantic content are too complex, resulting in too many label semantics in a single sentence. As a result, in the dynamic integrated attention voting system, the model does not pay enough attention to each label, which in turn affects the return probability of the label, making it difficult for the model to effectively capture the characteristics of each label; the third problem is the appearance of new labels, that is, the appearance of undefined new labels in the text. Problems of the first and second categories are manually eliminated or labeled by coders; text data for problems of the third category are manually encoded by coders. In addition, for the text data of the third category of problems, the implementation process of the multi-agent data enhancement stage in step S1 will be re-executed so that these data can be included in the coding book, thereby enriching the label library of the model.

[0081] During the model iteration phase, the process of step S2 is repeated to train and fine-tune each classifier and the dynamic integrated attention voting system based on the data in the new codebook. The iteratively trained codebook is then applied to the unlabeled text data. If there are still text data that fail recognition, but after analysis, it is determined that they do not fall into the third category, the text encoding can be considered saturated, and the overall process ends.

[0082] It should be noted that the number of representative sample selection trees in the manual coding stage in step S1 needs to be adjusted according to the actual requirements of the task, but the number of representative sample selection trees under each topic should not be less than 10. In the text generation process, the value range of Temperature is recommended to be 0.3-0.5, and this standard can be appropriately adjusted according to the actual task requirements. In the model training stage of step S2, appropriate classifiers and pre-training standards should be selected according to the actual characteristics of the data and the task requirements, but the number of classifiers should not be less than four. In the text coding and judgment stage of step S3, the coding conditions of the model need to be relaxed, and the maximum value coding requirements defined by formula (23) are no longer used. It is recommended to return all labels with a deviation range within 10% of the maximum value. This standard can also be appropriately adjusted according to the actual task requirements.

[0083] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A human-computer collaborative text data encoding method, the method comprising the following steps: Step S1: Create a code book: The process of formulating the codebook includes a coding text screening stage, a manual coding stage, and a multi-agent data enhancement stage. Step S2: Constructing a coding model: The construction of the encoding model is achieved through two stages: a model training stage and a dynamic integrated attention voting stage. Step S3: Verify the coding saturation: The verification of coding saturation includes two stages: a text coding and judgment stage and a model iteration stage.

2. The process of formulating a codebook according to claim 1, comprising a code text screening stage, a manual coding stage, and a multi-agent data enhancement stage. The specific contents of each stage are as follows: (1) Encoded text screening stage: Use clustering algorithms to reveal the potential topic structure of the corpus and calculate the probability distribution of each text on each topic. (2) Manual coding stage: Based on the probability matrix, the most representative samples are selected and multiple annotators are invited to annotate sentence by sentence in a back-to-back manner. At the same time, centralized discussion meetings are held according to the preset period to unify the coding results. (3) Multi-agent data enhancement stage: Utilize multiple large-scale language models to extract and refine each label and its semantic connotation to form standardized and operational label definitions; Subsequently, a multi-agent collaborative mechanism was introduced to carry out text generation and enhancement, and the sample size of each labeling system was balanced. The multi-agent system includes text generators, text analysts, and natural language training experts. The text generator needs to conduct a comprehensive review and analysis of the text under the label system, focusing on its language style, and rewrite the text according to the label definition, while implementing the modification suggestions of other agents. The responsibility of the text analyst is to rigorously judge and analyze the generated sentences based on the label definition to confirm whether the generated content remains within the label scope. If the generated content deviates from the label requirements, the analyst will provide specific and executable modification suggestions. During the evaluation process, the natural language training expert will combine the original text and label definition to determine whether the generated sentences introduce significant training noise. If noise is found, the expert will provide specific and executable modification suggestions.

3. The encoding model constructed according to claim 1 comprises two stages: a model training stage and a dynamic integrated attention voting stage. The rights at each stage are as follows: (1) Model training stage: Select four or more classifier models and train each classifier model based on the codebook formed in step S1. (2) Dynamic integrated attention voting stage: After each classifier model is trained, a classifier pool is constructed, and the judgment criteria for the model's "synergy ability" are constructed in the following way. The meaning of the parameters is detailed in the manual. Furthermore, the following formula is used as a standard to construct a voting system that combines the attention mechanism and Shapley weights. The meaning of the parameters is detailed in the specification: Attention l (x j )=softmax(W q ·h l (x j ))。 4. The verification of coding saturation according to claim 1 comprises a text coding and judgment phase and a model iteration phase. The specific contents of each phase are as follows: (1) Text encoding and judgment stage: When the collaborative ability of any classifier under a certain text data is less than 0, it is marked as recognition failure. The present invention proposes that the reasons for recognition failure include the following three categories: irrelevant topics, too complex texts, and the appearance of new labels. The first and second types of problems are manually eliminated or marked by coders; for text data of the third type of problems, coders manually encode them and then re-execute the implementation process of the multi-agent data enhancement stage in step S1, and include them in the coding book. (2) Model iteration phase: Repeat the operation process of step S2 to train and fine-tune each classifier and the dynamic integrated attention voting system based on the data in the new codebook. Use the iteratively trained code to re-encode the unlabeled text data. If all text data is recognized, it can be considered that the text encoding has reached saturation. If there is still unrecognizable text data, but it is determined that it does not belong to the third category of problems after judgment, it can be considered that the text encoding has reached saturation and the overall process is terminated.

Images