Data labeling method, electronic device, and program product
By combining vector knowledge bases with brief explanatory information, intent tags can be quickly filtered and accurately analyzed, solving the problem of low data annotation efficiency in operator customer service scenarios and achieving efficient and accurate data annotation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2026-02-13
- Publication Date
- 2026-06-05
AI Technical Summary
In the context of operator customer service, the demand for processing massive amounts of user dialogue data is increasing. The large number of intent categories leads to a surge in model training and rule maintenance costs. The diversity and ambiguity of user expressions can easily cause labeling errors. Furthermore, the dynamic changes in business rules require the system to have continuous optimization capabilities. Existing technologies have low data labeling efficiency.
By acquiring a vector knowledge base, vector retrieval is used to quickly filter out candidate intent tags. Combined with brief descriptions, intent analysis is performed to determine the target analysis results. Finally, the annotation information of the query dialogue is determined by comprehensively considering the vector retrieval, narrowing the intent matching range, and combining the targeted analysis of the tag descriptions to improve annotation efficiency and accuracy.
It effectively reduces subjective errors in manual annotation, improves data annotation efficiency, and ensures the accuracy and efficiency of intent judgment.
Smart Images
Figure CN122153440A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more particularly to a data annotation method, electronic device, and program product. Background Technology
[0002] In telecom operator customer service scenarios, the demand for processing massive amounts of user dialogue data is growing daily. Operators need to process millions of user-customer service conversations daily, covering hundreds of intent categories such as package inquiries, service applications, fault reporting, and complaints / suggestions. These conversations are complex and diverse, with users expressing themselves flexibly, and business rules are frequently updated.
[0003] Related technologies face multiple challenges when dealing with such scenarios: the large number of intent categories leads to a surge in model training and rule maintenance costs; the diversity and ambiguity of user expressions can easily cause labeling errors; and the dynamic changes in business rules require the system to have continuous optimization capabilities.
[0004] In the above process, the data annotation efficiency is low because the annotators usually read the dialogue content and select labels according to the specifications. Summary of the Invention
[0005] This application provides a data annotation method, electronic device, and program product to solve the problem of low efficiency in data annotation.
[0006] Firstly, this application provides a data annotation method, including:
[0007] Obtain the query dialogue to be labeled and the vector knowledge base. The vector knowledge base includes multiple intent tags, as well as brief descriptions and detailed descriptions for each intent tag.
[0008] Based on the vector knowledge base, the query dialogue is retrieved to determine multiple candidate intent tags;
[0009] For any candidate intent tag, perform intent analysis on the query dialogue based on the brief description of the candidate intent tag, and determine the target analysis result of the candidate intent tag.
[0010] Based on the target analysis results corresponding to multiple candidate intent tags, the annotation information of the query dialogue is determined.
[0011] In one possible implementation, intent analysis processing is performed on the query dialogue based on the brief description information of the candidate intent tags to determine the target analysis results of the candidate intent tags, including:
[0012] Determine the intent-based prompts, which include query dialogues, brief descriptions, and task instructions.
[0013] The intent judgment prompts are input into a preset large language model to obtain initial analysis results, which include the confidence scores of candidate intent labels.
[0014] The initial analysis results are verified for compliance, and if the verification passes, the initial analysis results are determined as the target analysis results.
[0015] In one possible implementation, the initial analysis results are subjected to compliance verification, and if the verification passes, the initial analysis results are determined as the target analysis results, including:
[0016] Determine whether the initial analysis results conform to the preset rules;
[0017] If so, the initial analysis result will be determined as the target analysis result;
[0018] If not, perform at least one retry operation until the number of retries is greater than or equal to the threshold, or the initial analysis results meet the preset rules.
[0019] In one possible implementation, the method further includes, before acquiring the query dialogue to be labeled and the vector knowledge base:
[0020] Obtain historical dialogue data corresponding to multiple intent tags;
[0021] For any given intent tag, generate detailed information about the intent tag based on its historical dialogue data.
[0022] The detailed description information is summarized and compressed to generate brief descriptions of intent tags;
[0023] Convert the detailed descriptions corresponding to multiple intent tags into multiple intent vectors;
[0024] A vector knowledge base is built based on multiple intent vectors. The vector knowledge base has a hierarchical index structure, which includes a first-level index and a second-level index.
[0025] In one possible implementation, the query dialogue is retrieved based on a vector knowledge base to determine multiple candidate intent tags, including:
[0026] The query dialogue is preprocessed and transformed into a vector to obtain a dialogue vector. The preprocessing includes at least one of the following: extracting key content and text normalization.
[0027] Using a similarity algorithm, multiple candidate intent tags are retrieved from the vector knowledge base. The similarity between the intent vector corresponding to the candidate intent tag and the dialogue vector is greater than or equal to the first threshold.
[0028] In one possible implementation, the method further includes:
[0029] Stratified sampling evaluation is performed on the labeled information of multiple query dialogues within a preset time period to obtain feedback data;
[0030] Based on the feedback data, the brief description or detailed description information corresponding to at least one intent tag in the vector knowledge base is updated.
[0031] In one possible implementation, for any intent tag, the brief description or detailed description information corresponding to the intent tag is updated based on feedback data, including:
[0032] Based on the feedback data, determine whether the preset triggering conditions corresponding to the intent tag are met;
[0033] If so, identify the incorrectly labeled samples, correctly labeled samples, and feedback information corresponding to the intent label; based on the incorrectly labeled samples, correctly labeled samples, and feedback information, determine the reason for the labeling error corresponding to the intent label.
[0034] Based on the reason for the labeling error, generate update description information, which includes detailed updated information and brief updated information.
[0035] The update description information is validated, and after validation, the description information and intent vector of the corresponding intent tag in the vector knowledge base are updated according to the update description information.
[0036] Secondly, this application provides a data annotation device, comprising:
[0037] The acquisition module is used to acquire the query dialogue to be labeled and the vector knowledge base. The vector knowledge base includes multiple intent tags, as well as brief description information and detailed description information corresponding to each intent tag.
[0038] The determination module is used to retrieve query dialogues based on a vector knowledge base and determine multiple candidate intent tags;
[0039] The analysis module is used to perform intent analysis on the query dialogue based on the brief description information of any candidate intent tag, and to determine the target analysis result of the candidate intent tag.
[0040] The annotation module is used to determine the annotation information of the query dialogue based on the target analysis results corresponding to multiple candidate intent tags.
[0041] In one possible implementation, the analysis module is specifically used for:
[0042] Determine the intent-based prompts, which include query dialogues, brief descriptions, and task instructions.
[0043] The intent judgment prompts are input into a preset large language model to obtain initial analysis results, which include the confidence scores of candidate intent labels.
[0044] The initial analysis results are verified for compliance, and if the verification passes, the initial analysis results are determined as the target analysis results.
[0045] In one possible implementation, the analysis module is specifically used for:
[0046] Determine whether the initial analysis results conform to the preset rules;
[0047] If so, the initial analysis result will be determined as the target analysis result;
[0048] If not, perform at least one retry operation until the number of retries is greater than or equal to the threshold, or the initial analysis results meet the preset rules.
[0049] In one possible implementation, the apparatus further includes a generation module, which is used to:
[0050] Obtain historical dialogue data corresponding to multiple intent tags;
[0051] For any given intent tag, generate detailed information about the intent tag based on its historical dialogue data.
[0052] The detailed description information is summarized and compressed to generate brief descriptions of intent tags;
[0053] Convert the detailed descriptions corresponding to multiple intent tags into multiple intent vectors;
[0054] A vector knowledge base is built based on multiple intent vectors. The vector knowledge base has a hierarchical index structure, which includes a first-level index and a second-level index.
[0055] In one possible implementation, the determining module is specifically used for:
[0056] The query dialogue is preprocessed and transformed into a vector to obtain a dialogue vector. The preprocessing includes at least one of the following: extracting key content and text normalization.
[0057] Using a similarity algorithm, multiple candidate intent tags are retrieved from the vector knowledge base. The similarity between the intent vector corresponding to the candidate intent tag and the dialogue vector is greater than or equal to the first threshold.
[0058] In one possible implementation, the apparatus further includes an update module, which is used to:
[0059] Stratified sampling evaluation is performed on the labeled information of multiple query dialogues within a preset time period to obtain feedback data;
[0060] Based on the feedback data, the brief description or detailed description information corresponding to at least one intent tag in the vector knowledge base is updated.
[0061] In one possible implementation, for any intent tag, the update module is specifically used to:
[0062] Based on the feedback data, determine whether the preset triggering conditions corresponding to the intent tag are met;
[0063] If so, identify the incorrectly labeled samples, correctly labeled samples, and feedback information corresponding to the intent label; based on the incorrectly labeled samples, correctly labeled samples, and feedback information, determine the reason for the labeling error corresponding to the intent label.
[0064] Based on the reason for the labeling error, generate update description information, which includes detailed updated information and brief updated information.
[0065] The update description information is validated, and after validation, the description information and intent vector of the corresponding intent tag in the vector knowledge base are updated according to the update description information.
[0066] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0067] The memory stores the instructions that the computer executes;
[0068] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.
[0069] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.
[0070] Fifthly, this application provides a computer program product, including a computer program that, when executed by a computer, implements the method as described in any of the first aspects.
[0071] This application provides a data annotation method, electronic device, and program product. It acquires a query dialogue to be annotated and a vector knowledge base, which includes multiple intent tags and brief and detailed descriptions for each intent tag. Based on the vector knowledge base, the query dialogue is retrieved to determine multiple candidate intent tags. For any candidate intent tag, intent analysis is performed on the query dialogue based on the brief descriptions of the candidate intent tag to determine the target analysis result of the candidate intent tag. Based on the target analysis results corresponding to the multiple candidate intent tags, the annotation information for the query dialogue is determined. This approach not only rapidly narrows the intent matching range through vector retrieval, improving the efficiency of query dialogue intent annotation, but also ensures the accuracy of intent judgment through targeted intent analysis based on tag brief descriptions, effectively reducing subjective errors in manual annotation and improving the efficiency of data annotation. Attached Figure Description
[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0073] Figure 1 This is a schematic diagram of the architecture of a data annotation system provided in an embodiment of this application;
[0074] Figure 2 A flowchart illustrating a data annotation method provided in an embodiment of this application;
[0075] Figure 3 A flowchart illustrating another data annotation method provided in an embodiment of this application;
[0076] Figure 4 A flowchart illustrating yet another data annotation method provided in an embodiment of this application;
[0077] Figure 5 This is a schematic diagram of the structure of a data annotation device provided in an embodiment of this application;
[0078] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0079] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0080] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.
[0082] It should be noted that the data annotation method, electronic device and program product provided in this application can be used in the field of data processing, or in any field other than data processing. The application field of the data annotation method, electronic device and program product in this application is not limited.
[0083] In telecom operator customer service scenarios, the demand for processing massive amounts of user dialogue data is growing daily. Operators need to process millions of user-customer service conversations daily, covering hundreds of intent categories such as package inquiries, service applications, fault reporting, and complaints / suggestions. These conversations are complex and diverse, with users expressing themselves flexibly, and business rules are frequently updated.
[0084] Related technologies face multiple challenges when dealing with such scenarios: the large number of intent categories leads to a surge in model training and rule maintenance costs; the diversity and ambiguity of user expressions can easily cause labeling errors; and the dynamic changes in business rules require the system to have continuous optimization capabilities.
[0085] In the above process, the data annotation efficiency is low because the annotators usually read the dialogue content and select labels according to the specifications.
[0086] To address the aforementioned technical issues, this application provides a data annotation method. By using similarity retrieval based on a vectorized intent knowledge base, a small number of the most relevant candidate intent tags are quickly and efficiently selected from the full set of tags. Then, for each candidate intent tag, intent analysis of the query dialogue is performed based on its brief description, yielding corresponding target analysis results. Finally, the annotation information for the query dialogue is determined comprehensively based on the target analysis results of each candidate tag. This approach not only rapidly narrows the intent matching range using vector retrieval, improving the efficiency of query dialogue intent annotation, but also ensures the accuracy of intent judgment through targeted intent analysis based on tag brief descriptions, effectively reducing subjective errors in manual annotation and improving the efficiency of data annotation.
[0087] Below, in conjunction with Figure 1 The architecture of the data annotation system will be explained.
[0088] Figure 1 This is a schematic diagram of the architecture of a data annotation system provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 , Figure 1 It may include a data annotation system, which may include a description generation unit, a retrieval unit, a judgment unit, a model feedback unit, and an iterative optimization unit.
[0089] Each unit independently implements its core functions, and forms a closely connected whole in terms of data flow and business processes, constituting a complete annotation closed loop.
[0090] The description generation unit can be used to automatically generate and standardize descriptions related to intent categories based on historical real dialogue data in the operator's customer service dialogue database.
[0091] This unit first aggregates and preprocesses the labeled historical dialogue data, and obtains standardized data through text cleaning, sensitive information desensitization, and dialogue fragment extraction. Then, it calls a large language model to generate structured detailed descriptions and concise descriptions for each intent category. After proofreading and revision, it forms standardized text information containing intent tags, corresponding concise descriptions, and detailed descriptions.
[0092] The retrieval unit can be used to quickly and accurately narrow down the matching range from the full set of intent tags, and provides a target candidate set for subsequent judgment units.
[0093] This unit can first embed the detailed intent description output by the description generation unit into text and construct a vector index to form a vector knowledge base containing intent tags and corresponding description information.
[0094] This unit can also extract fragments, standardize and vectorize the dialogue text after receiving the query dialogue to be labeled, and retrieve and determine multiple candidate intent tags most relevant to the query dialogue to be labeled from the vector knowledge base.
[0095] The judgment unit can be used to perform refined intent matching judgment on the candidate intent tags filtered by the retrieval unit, and finally determine the initial annotation information of the query dialogue.
[0096] This unit can construct structured information based on the candidate intent tags output by the retrieval unit, including task description, dialogue content, concise description of candidate tags, judgment requirements, and standardized output format. It calls a large language model to combine the concise description information of candidate intent tags to perform intent analysis processing on the query dialogue, completes the matching degree judgment of candidate tags one by one, and outputs the corresponding target analysis results.
[0097] This unit can perform multi-dimensional analysis and verification of the model output results, including format, label ID, confidence level, and quantity. It executes an automatic retry mechanism for verification failures and finally generates initial annotation results containing dialogue ID, annotation labels, confidence level, and other information. These results are synchronously written to the database and pushed to the subsequent model feedback unit.
[0098] The model feedback unit can be used as a key bridge connecting the initial annotation results of the judgment unit and the system optimization of the iterative optimization unit.
[0099] This unit can use a stratified sampling method based on confidence level to screen samples based on the automatic labeling results output by the judgment unit, ensuring sample coverage for each intent label.
[0100] This unit can also perform operations such as confirming the correctness of the sampled samples, modifying labels, adding labels, and providing feedback. It collects relevant feedback on the labeling quality and explanatory content, and then standardizes the feedback data to construct positive and negative samples, mine difficult samples, summarize explanatory issues, and prioritize optimization to form a structured feedback dataset. This provides a clear optimization direction and core data support for the iterative optimization unit.
[0101] The iterative optimization unit can be used to optimize the intent category description and update the vector knowledge base based on the structured feedback data from the model feedback unit, thereby promoting the continuous improvement of the system's annotation capabilities.
[0102] This unit can be set to optimize multiple dimensions such as periodicity, error rate, feedback volume, and new tags, and the optimization process will be started when any condition is met.
[0103] This unit can perform error case analysis on the tags to be optimized, count error types, identify easily confused tags, cluster error samples and locate problems in the description content, and then call a large language model to generate optimized intent category descriptions. The optimal optimized version is selected through comparative learning strategies, and the optimization effect is verified through offline verification, testing and review by business experts.
[0104] Furthermore, after verification, this unit can synchronize the optimized description content to the description generation unit, update the vector knowledge base and rebuild the vector index, and at the same time fully record the optimization history information, so as to realize the system's self-learning and self-optimization, and continuously improve the collaborative work effect of each unit as the business progresses.
[0105] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0106] Figure 2 This is a flowchart illustrating a data annotation method provided in an embodiment of this application. The execution entity in this embodiment can be a server or a processor within the server. The processor can be implemented in software or a combination of software and hardware. Please refer to... Figure 2 The method includes:
[0107] S201. Obtain the query dialogue and vector knowledge base to be labeled.
[0108] The vector knowledge base includes multiple intent tags, as well as brief and detailed descriptions for each intent tag.
[0109] Intent tags can be standardized identifiers used to represent the core intent of a query dialogue.
[0110] Brief descriptions can be used to refine and distill feature descriptions.
[0111] Detailed information can be used for comprehensive structured feature descriptions.
[0112] For example, a brief description can be around 200 words, including the core definition of intent, typical user expressions, key business elements, and other information.
[0113] The detailed description can be around 500 words, including the intent definition, typical scenarios, common expressions, business elements, and differentiation of similar intents.
[0114] The query dialogues to be labeled may refer to those originating from scenarios such as operator customer service, and have not been labeled with intent tags.
[0115] Vector knowledge bases can be used to store intent tags and corresponding descriptive information, and support fast retrieval and matching.
[0116] Unlabeled interactive dialogues and a pre-built vector knowledge base can be extracted from the operator's customer service dialogue database through API calls, database reads, and other methods. During the extraction process, metadata such as dialogue identifiers, interaction times, and user identifiers are associated to ensure that each dialogue to be labeled is uniquely traceable. At the same time, the extracted dialogues are initially filtered to remove meaningless data such as blank dialogues and invalid garbled dialogues.
[0117] The vector knowledge base can be generated and maintained by the specification generation unit.
[0118] The vector knowledge base can also store the vector representations and associated metadata of each intent tag, supporting fast retrieval and retrieval.
[0119] Optionally, when the amount of dialogue data to be labeled is large and the vector knowledge base is stored in a distributed database, a distributed acquisition strategy can be adopted: the dialogue data to be labeled is extracted in parallel through distributed crawlers or multiple interfaces, and the extracted data is stored in shards; the vector knowledge base can be accessed through distributed index calls, and can be obtained in shards according to business categories, reducing the pressure of single interface calls and improving the data acquisition speed.
[0120] Optionally, for real-time annotation scenarios, the query dialogues to be annotated can be incrementally obtained through a real-time stream processing framework, and each unannotated dialogue is pushed to the annotation process in real time; the vector knowledge base can adopt an incremental update mechanism, and when a new intent tag is added or the original tag description is optimized, it is updated in real time to the available knowledge base for retrieval, ensuring that the obtained knowledge base is always the latest version and avoiding annotation errors caused by knowledge base lag.
[0121] Optionally, before obtaining the query dialogues to be labeled and the vector knowledge base, the method further includes establishing a vector knowledge base: obtaining historical dialogue data corresponding to multiple intent tags respectively; for any intent tag, generating detailed description information of the intent tag based on the historical dialogue data of the intent tag; performing summary compression processing on the detailed description information to generate brief description information of the intent tag; converting the detailed description information corresponding to multiple intent tags into multiple intent vectors; and establishing a vector knowledge base based on the multiple intent vectors.
[0122] The vector knowledge base has a hierarchical index structure, which includes a first-level index and a second-level index.
[0123] From business scenario databases such as operator customer service dialogue databases, we can extract historical dialogue data that has been manually annotated, classify and aggregate it according to intent tags, and ensure that each intent tag corresponds to a dedicated historical dialogue dataset.
[0124] Historical dialogue data can ensure the sample coverage of each dataset, covering typical business scenarios and common user expressions corresponding to the intent tag. Furthermore, the extracted historical dialogue data is initially filtered to remove invalid garbled characters, blank content, and redundant dialogue fragments unrelated to the intent, while retaining effective dialogue data that is highly relevant to the intent tag.
[0125] For a single intent tag corresponding to valid historical dialogue data, a large language model can be invoked and structured prompt words can be combined to generate detailed information about the intent tag from dimensions such as intent definition, typical scenarios, common expressions, business elements, and differentiation of similar intents.
[0126] Based on the detailed description information of the intent tag, the detailed description information can be simplified by combining large language model summarization or keyword extraction with text reconstruction, retaining content such as the core definition of the intent, typical user expressions, and key business elements, to generate a brief description of the intent tag.
[0127] A pre-trained Chinese text embedding model can be used to perform text vectorization on the detailed description information corresponding to each intent tag, mapping the structured text information into a high-dimensional numerical intent vector.
[0128] During vectorization, the consistency of the embedding model is ensured, guaranteeing that all detailed information of intent labels is converted into intent vectors of the same dimension and semantic space, thus avoiding vector matching deviations caused by model differences.
[0129] Based on all generated intent vectors, a hierarchical indexed vector knowledge base with first-level and second-level indexes can be built using a vector database. The first-level index is a coarse-grained business category index, dividing all intent tags into business categories such as query, processing, fault reporting, and complaint / suggestion, and creating a dedicated index partition for each business category to achieve preliminary classification of all intent vectors. The second-level index is a fine-grained sub-category index, creating a refined index for all intent vectors within each business category's index partition to achieve precise location of intent vectors.
[0130] S202. Based on the vector knowledge base, retrieve the query dialogue and determine multiple candidate intent tags.
[0131] Candidate intent tags can be selected from the full set of intent tags in the vector knowledge base, and are potentially relevant to the query dialogue to be labeled.
[0132] It can standardize the query dialogue to be labeled, and can use a preset embedding model to convert the preprocessed query dialogue to be labeled into a dialogue vector representation. It can calculate the similarity between the dialogue vector and each intent tag vector in the knowledge base by using cosine similarity, and filter out the top K intent tags with the highest similarity to determine multiple candidate intent tags.
[0133] The standardization process may include using a sliding window method to extract key segments of the dialogue, prioritizing the retention of user problem descriptions and demand statements, and filtering out repeated confirmation statements from customer service representatives; unifying simplified and traditional Chinese characters through preset tools, correcting common typos using a pinyin edit distance algorithm, expanding abbreviations with the help of a predefined dictionary, and removing colloquial filler words using a stop word list, in order to ensure the standardization of the dialogue text.
[0134] The preset embedding model can be consistent with the embedding model used when building the vector knowledge base, ensuring that the semantic space of the dialogue vectors and the intent tag vectors in the knowledge base is consistent, thereby improving the accuracy of similarity matching.
[0135] Optionally, multiple candidate intent tags can be determined by retrieving the query dialogue from the vector knowledge base in the following way: preprocessing the query dialogue with text and performing vector transformation to obtain the dialogue vector; retrieving multiple candidate intent tags from the vector knowledge base using a similarity algorithm.
[0136] Among them, the similarity between the intent vector corresponding to the candidate intent label and the dialogue vector is greater than or equal to the first threshold.
[0137] Text preprocessing includes at least one of the following: extracting key content and text normalization.
[0138] Extracting key content can be achieved using any one or more combinations of sliding window method, keyword matching method, or semantic extraction method, extracting content directly related to the user's intent from the complete query dialogue.
[0139] Text normalization can standardize and organize the extracted key content, eliminate differences caused by text format and expression habits, and ensure that the processed dialogue text is semantically consistent and formatted correctly.
[0140] Commonly used vector similarity measurement algorithms such as cosine similarity algorithm and Euclidean distance algorithm can be selected for similarity evaluation.
[0141] A first threshold can be preset, and the similarity between all intent vectors and dialogue vectors in the vector knowledge base can be compared. Intent vectors with a similarity greater than or equal to the first threshold are selected. The intent tags corresponding to these intent vectors are the candidate intent tags related to the query dialogue to be labeled.
[0142] This threshold can be flexibly adjusted according to business needs and label size; for example, it can be set to 0.6.
[0143] S203. For any candidate intent tag, perform intent analysis processing on the query dialogue based on the brief description information of the candidate intent tag, and determine the target analysis result of the candidate intent tag.
[0144] The target analysis results can be an evaluation of the matching degree between a single candidate intent label and the query dialogue to be labeled.
[0145] The results of the target analysis can include information such as the confidence level of the match and the reason for the match.
[0146] The results of the target analysis can be used to support the determination of the final annotation information.
[0147] For any candidate intent tag, intent judgment prompts can be determined based on the brief description of the candidate intent tag. Based on the intent judgment prompts, intent analysis processing can be performed on the query dialogue to determine the target analysis result of the candidate intent tag.
[0148] The intent judgment prompts may include task descriptions, dialogue content, brief descriptions of candidate intent tags, judgment requirements, and output format.
[0149] The task description can be used to explicitly require the model to determine the degree of match between a single candidate intent label and the query dialogue to be labeled.
[0150] The dialogue content can be key segments of the dialogue to be annotated and queried.
[0151] The brief description information corresponding to the candidate intent label can avoid the intent judgment prompts from being too long due to detailed information, which would exceed the model context window limit. At the same time, the brief description already contains the most critical recognition features.
[0152] The judgment requirements can be to ask the model to output a conclusion on whether it matches or not, a confidence score in the 0-1 interval, and specific reasons for the match.
[0153] The output format can be a specified preset format to ensure that the analysis results can be quickly parsed by subsequent steps.
[0154] S204. Based on the target analysis results corresponding to multiple candidate intent tags, determine the annotation information of the query dialogue.
[0155] The annotation information can be used to clarify the intent category corresponding to the query dialogue to be annotated.
[0156] The annotation information can include structured information such as dialogue identifiers, matched intent tags, confidence levels of each tag, and matching reasons.
[0157] All target analysis results corresponding to all candidate intent tags can be obtained, sorted in descending order according to the confidence score, the candidate tags with a confidence of 0 are removed, the valid analysis results are retained, a threshold N for the number of labeled tags is set, and candidate tags with a confidence ≥ 0.8 can be preferentially selected; if the number of high-confidence tags ≥ N, then the top N tags with the highest confidence are selected as matching tags; if the number of high-confidence tags < N, then medium-confidence tags with a confidence in the range [0.5, 0.8) are additionally selected until N tags are reached; if the number of all valid tags < N, then the actual number of valid tags is selected, and according to the screened matching tags, annotation information is generated.
[0158] Among them, the annotation information may include the conversation identifier, the matching tag, the confidence of each tag, the reason for matching, the list of candidate tags, the processing time, and so on.
[0159] Optionally, the method further includes an update: performing stratified sampling evaluation on the annotation information of multiple query conversations within a preset duration to obtain feedback data; according to the feedback data, performing update processing on the brief description information or detailed description information corresponding to at least one intent tag.
[0160] In this way, by collecting feedback through sampling evaluation of the automatic annotation results and then optimizing the description information of the intent tags according to the feedback, the accuracy of the annotation system is continuously improved with actual use.
[0161] A data annotation method provided in this embodiment, by obtaining a query conversation to be annotated and a vector knowledge base, the vector knowledge base includes multiple intent tags, as well as the brief description information and detailed description information corresponding to each intent tag; according to the vector knowledge base, retrieving the query conversation to determine multiple candidate intent tags; for any one candidate intent tag, according to the brief description information of the candidate intent tag, performing intent analysis processing on the query conversation to determine the target analysis result of the candidate intent tag; according to the target analysis results respectively corresponding to multiple candidate intent tags, determining the annotation information of the query conversation. In this way, not only can the intent matching range be quickly narrowed by vector retrieval to improve the efficiency of query conversation intent annotation, but also the accuracy of intent judgment can be ensured through targeted intent analysis based on the brief description of the tag, effectively reducing the subjective error of manual annotation and improving the efficiency of data annotation.
[0162] Next, in combination with Figure 3 , the process of performing intent analysis processing on the query conversation according to the brief description information of the candidate intent tag to determine the target analysis result of the candidate intent tag will be explained.
[0163] Figure 3 It is a schematic flowchart of another data annotation method provided in an embodiment of the present application. Please refer to Figure 3 This method includes:
[0164] S301. Determine the intent and judge the prompt information.
[0165] Intent-based prompts include query dialogues, brief descriptions, and task instructions.
[0166] Intent judgment prompts are input information that guides the large language model to perform correct analysis and judgment.
[0167] The query dialogue can be the original dialogue text to be annotated, or a pre-processed key segment of the dialogue.
[0168] For example, the task instruction could be: "You are an experienced operator customer service expert. Based on the following dialogue and the description of each candidate intent, please select the 1st (or Nth) intent label that best matches."
[0169] Based on the structured format, the intent-based prompts can be determined.
[0170] S302. Input the intent judgment prompt information into the preset large language model to obtain the initial analysis results.
[0171] The initial analysis results include the confidence scores of the candidate intent labels.
[0172] For example, a large language model can play the role of a "customer service domain expert" based on its massive pre-trained knowledge and understanding of prompts.
[0173] Large language models can compare the labeled dialogue with the concise description of each candidate intent, analyze the degree of matching between user needs, context and various intent definitions, typical scenarios, keywords and other elements in the dialogue, and perform comprehensive reasoning.
[0174] Intent judgment prompts can be input into a preset large language model, which will then generate structured initial analysis results based on the reasoning process.
[0175] S303. Perform compliance verification on the initial analysis results, and if the verification passes, determine the initial analysis results as the target analysis results.
[0176] Compliance verification can be a series of automated checks performed by the system on the initial analysis results returned by the model.
[0177] Compliance verification can include format verification, label validity verification, confidence range verification, logical verification, and quantity verification, etc.
[0178] Format validation can verify whether the output is in a valid JSON format.
[0179] Label validity verification can confirm that all label identifiers in the output result exist in the candidate intent label list of this input.
[0180] Confidence range verification can check whether all confidence values are within a preset range.
[0181] Logical verification can check whether the sum of the confidence scores of each label exceeds a reasonable upper limit, thus preventing abnormal probability distribution.
[0182] Quantity verification can be used to check whether the number of output tags does not exceed the maximum number N specified in the task instruction.
[0183] The initial analysis results can be validated for compliance. If all the above validations pass, the initial analysis results are considered compliant and are officially determined as the target analysis results for this processing. If the validation fails, the exception handling mechanism can be activated.
[0184] Optionally, the initial analysis results can be validated for compliance in the following way, and the initial analysis results can be determined as the target analysis results when the validation is successful: determine whether the initial analysis results meet the preset rules; if yes, determine the initial analysis results as the target analysis results; if no, perform at least one retry operation until the number of retry operations is greater than or equal to the threshold, or the initial analysis results meet the preset rules.
[0185] The retry operation can involve inputting intent judgment prompts into a preset large language model to obtain initial analysis results, and then performing compliance verification on the initial analysis results.
[0186] The implementation details of each step in this application embodiment can be found in the description of the corresponding steps or operations in the above method embodiments; repeated content will not be repeated.
[0187] This embodiment provides a data annotation method that determines intent-based prompts, including query dialogues, brief descriptions, and task instructions. These prompts are then input into a pre-defined large language model to obtain initial analysis results, which include the confidence levels of candidate intent tags. The initial analysis results undergo compliance verification, and upon successful verification, are designated as the target analysis results. This approach guides the large language model to perform focused analysis through structured prompts, and combined with an automated verification mechanism, ensures the reliability and structure of the final output. It provides directly usable, confident judgment criteria for high-quality automated annotation, thereby improving the efficiency of data annotation.
[0188] Below, in conjunction with Figure 4 The process of generating rendering data based on the target task and the target 3D model is explained.
[0189] Figure 4 This is a flowchart illustrating another data annotation method provided in an embodiment of this application. Based on the above embodiments, please refer to... Figure 4 .
[0190] S401. Perform stratified sampling evaluation on the labeled information of multiple query dialogues within a preset time period to obtain feedback data.
[0191] You can select the annotation information of all query dialogues that are automatically annotated within a preset time period, divide the samples into different levels according to the confidence score in the annotation information, set differentiated sampling ratios for different levels, and generate feedback data.
[0192] For example, 5% of the samples are taken from the high-confidence layer (confidence ≥ 0.8, high labeling reliability); 20% of the samples are taken from the medium-confidence layer (0.5 ≤ confidence < 0.8, labeling results are questionable); and 50% of the samples are taken from the low-confidence layer (confidence < 0.5, high probability of labeling error). At the same time, it is ensured that samples are drawn for each intent label to avoid the problem of missing niche labels.
[0193] Feedback data can include: details of correctly / incorrectly labeled samples, error types (omission / mislabeling / confusion, etc.), a list of error-prone intent labels, specific problem descriptions of label descriptions, and reference samples with correct labeling, providing a clear basis for subsequent optimization.
[0194] S402. Based on the feedback data, determine whether the preset triggering conditions corresponding to the intent tag are met.
[0195] Preset trigger conditions can include periodic trigger conditions, error rate trigger conditions, feedback volume trigger conditions, new tag trigger conditions, etc.
[0196] Error rate triggering conditions can refer to the labeling error rate exceeding a preset threshold during sampling evaluation.
[0197] Feedback triggers can be triggered when the accumulated feedback for the label reaches a certain number.
[0198] Periodic triggering can refer to routinely reviewing and optimizing tags at a fixed optimization cycle, regardless of their current performance.
[0199] New tag triggering can refer to the initial optimization of a newly added intent tag after it has been used a certain number of times, in order to calibrate its description.
[0200] If the triggering conditions for a specific intent tag are met, the optimization process is initiated for that tag.
[0201] For each intent tag, its related feedback data can be analyzed, and it can be determined whether any of the following conditions are met: periodic triggering condition, error rate triggering condition, feedback volume triggering condition, or new tag triggering condition.
[0202] S403. If so, determine the incorrectly labeled samples, correctly labeled samples, and feedback information corresponding to the intent label, and determine the reason for the labeling error corresponding to the intent label based on the incorrectly labeled samples, correctly labeled samples, and feedback information.
[0203] The following analytical methods can be used to determine the reasons for labeling errors corresponding to intent tags:
[0204] Statistical distribution of error types;
[0205] Generate a confusion matrix to accurately identify other intent tags that are most likely to be confused with this intent;
[0206] Clustering algorithms are used to cluster erroneous samples to discover common user expressions or scenario features;
[0207] By comparing the current description with the error samples and correct samples, specific paragraphs in the description that are vague, lack key scenes, or have insufficient distinguishability can be identified.
[0208] S404. Generate update description information based on the reason for the annotation error.
[0209] The update notes include both detailed and brief descriptions of the update.
[0210] Multiple optimized versions can be generated based on the reasons for the labeled errors. Each optimized version is scored by an automatic scoring model based on three dimensions: distinguishability from erroneous samples, consistency with correct samples, and clarity and completeness of the text. The version with the highest score is selected as the update description information.
[0211] S405. Perform effect verification processing on the update description information, and after the verification is passed, update the description information and intent vector of the corresponding intent tag in the vector knowledge base according to the update description information.
[0212] Performance verification can include offline backtesting, etc.
[0213] The updated description information is validated. After validation, the detailed description and brief description of the corresponding intent tag in the vector knowledge base are replaced with the updated description information. Using the same embedding model, the new detailed description information is converted into a new intent vector. The version number is incremented for the description information of the intent tag, and a complete optimization log is recorded.
[0214] The implementation details of each step in this application embodiment can be found in the description of the corresponding steps or operations in the above method embodiments; repeated content will not be repeated.
[0215] This embodiment provides a data annotation method that, based on feedback data, determines whether the preset triggering conditions corresponding to the intent tag are met. If so, it identifies the erroneous annotation samples, correct annotation samples, and feedback information corresponding to the intent tag. Based on these, it determines the cause of the annotation error for the intent tag. According to the cause of the annotation error, it generates update description information, including updated detailed description information and updated brief description information. The update description information is then validated, and upon successful validation, the description information and intent vector of the corresponding intent tag in the vector knowledge base are updated based on the update description information. In this way, by establishing a closed-loop optimization mechanism based on feedback data automatic triggering, accurate analysis of error root causes, and validation, the core knowledge base of the system can adaptively iterate and improve, thereby continuously improving the accuracy and robustness of subsequent automatic annotation from the root, and increasing the efficiency of data annotation.
[0216] Figure 5 This is a schematic diagram of a data annotation device provided in an embodiment of this application. Please refer to... Figure 5 The data annotation device 500 includes an acquisition module 501, a determination module 502, an analysis module 503, and an annotation module 504.
[0217] The acquisition module 501 is used to acquire the query dialogue to be labeled and the vector knowledge base. The vector knowledge base includes multiple intent tags, as well as brief description information and detailed description information corresponding to each intent tag.
[0218] The determination module 502 is used to retrieve the query dialogue based on the vector knowledge base and determine multiple candidate intent tags;
[0219] Analysis module 503 is used to perform intent analysis processing on the query dialogue based on the brief description information of the candidate intent tag for any candidate intent tag, and determine the target analysis result of the candidate intent tag.
[0220] The annotation module 504 is used to determine the annotation information of the query dialogue based on the target analysis results corresponding to multiple candidate intent tags.
[0221] In one possible implementation, the analysis module 503 is specifically used for:
[0222] Determine the intent-based prompts, which include query dialogues, brief descriptions, and task instructions.
[0223] The intent judgment prompts are input into a preset large language model to obtain initial analysis results, which include the confidence scores of candidate intent labels.
[0224] The initial analysis results are verified for compliance, and if the verification passes, the initial analysis results are determined as the target analysis results.
[0225] In one possible implementation, the analysis module 503 is specifically used for:
[0226] Determine whether the initial analysis results conform to the preset rules;
[0227] If so, the initial analysis result will be determined as the target analysis result;
[0228] If not, perform at least one retry operation until the number of retries is greater than or equal to the threshold, or the initial analysis results meet the preset rules.
[0229] In one possible implementation, the apparatus further includes a generation module 505, which is configured to:
[0230] Obtain historical dialogue data corresponding to multiple intent tags;
[0231] For any given intent tag, generate detailed information about the intent tag based on its historical dialogue data.
[0232] The detailed description information is summarized and compressed to generate brief descriptions of intent tags;
[0233] Convert the detailed descriptions corresponding to multiple intent tags into multiple intent vectors;
[0234] A vector knowledge base is built based on multiple intent vectors. The vector knowledge base has a hierarchical index structure, which includes a first-level index and a second-level index.
[0235] In one possible implementation, the determining module 502 is specifically used for:
[0236] The query dialogue is preprocessed and transformed into a vector to obtain a dialogue vector. The preprocessing includes at least one of the following: extracting key content and text normalization.
[0237] Using a similarity algorithm, multiple candidate intent tags are retrieved from the vector knowledge base. The similarity between the intent vector corresponding to the candidate intent tag and the dialogue vector is greater than or equal to the first threshold.
[0238] In one possible implementation, the device further includes an update module 506, which is configured to:
[0239] Stratified sampling evaluation is performed on the labeled information of multiple query dialogues within a preset time period to obtain feedback data;
[0240] Based on the feedback data, the brief description or detailed description information corresponding to at least one intent tag in the vector knowledge base is updated.
[0241] In one possible implementation, for any intent tag, the update module 506 is specifically used for:
[0242] Based on the feedback data, determine whether the preset triggering conditions corresponding to the intent tag are met;
[0243] If so, identify the incorrectly labeled samples, correctly labeled samples, and feedback information corresponding to the intent label; based on the incorrectly labeled samples, correctly labeled samples, and feedback information, determine the reason for the labeling error corresponding to the intent label.
[0244] Based on the reason for the labeling error, generate update description information, which includes detailed updated information and brief updated information.
[0245] The update description information is validated, and after validation, the description information and intent vector of the corresponding intent tag in the vector knowledge base are updated according to the update description information.
[0246] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 6 The electronic device 600 may include: a memory 601, a processor 602, and a transceiver 603.
[0247] Memory 601 is used to store program instructions;
[0248] The processor 602 is used to execute the program instructions stored in the memory so that the electronic device 600 performs the above-described method.
[0249] Transceiver 603 may include a transmitter and / or a receiver. The transmitter may also be referred to as a transmitter, transmitter port, or transmitter interface, and the receiver may also be referred to as a receiver port, receiver interface, or similar descriptions. Exemplarily, memory 601, processor 602, and transceiver 603 are interconnected via bus 604.
[0250] This application also provides a computer program product that can be executed by a processor, and when the computer program product is executed, the above-described method can be implemented.
[0251] The data annotation device, electronic device, computer-readable storage medium, and computer program product of the embodiments of this application can execute the technical solutions shown in the above method embodiments. Their implementation principles and beneficial effects are similar, and will not be described again here.
[0252] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0253] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0254] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0255] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0256] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0257] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0258] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0259] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0260] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A data annotation method, characterized in that, The method includes: Obtain the query dialogue to be labeled and the vector knowledge base, wherein the vector knowledge base includes multiple intent tags, as well as brief description information and detailed description information corresponding to each intent tag; Based on the vector knowledge base, the query dialogue is retrieved to determine multiple candidate intent tags; For any candidate intent tag, based on the brief description information of the candidate intent tag, perform intent analysis processing on the query dialogue to determine the target analysis result of the candidate intent tag; Based on the target analysis results corresponding to the multiple candidate intent tags, the annotation information of the query dialogue is determined.
2. The method according to claim 1, characterized in that, Based on the brief description information of the candidate intent tags, intent analysis processing is performed on the query dialogue to determine the target analysis result of the candidate intent tags, including: Determine the intent judgment prompt information, which includes the query dialogue, the brief description, and the task instruction; The intent judgment prompt information is input into a preset large language model to obtain initial analysis results, which include the confidence level of the candidate intent label; The initial analysis results are subjected to compliance verification, and if the verification passes, the initial analysis results are determined as the target analysis results.
3. The method according to claim 2, characterized in that, Perform compliance verification on the initial analysis results, and if the verification passes, determine the initial analysis results as the target analysis results, including: Determine whether the initial analysis results conform to preset rules; If so, the initial analysis result shall be determined as the target analysis result; If not, at least one retry operation will be performed until the number of retry operations is greater than or equal to the threshold, or the initial analysis result meets the preset rules.
4. The method according to any one of claims 1-3, characterized in that, Before acquiring the query dialogue and vector knowledge base to be labeled, the method further includes: Obtain historical dialogue data corresponding to multiple intent tags; For any given intent tag, generate detailed information about the intent tag based on its historical dialogue data. The detailed description information is summarized and compressed to generate a brief description of the intent tag; The detailed descriptions corresponding to the multiple intent tags are converted into multiple intent vectors; Based on the multiple intent vectors, a vector knowledge base is established. The vector knowledge base has a hierarchical index structure, which includes a first-level index and a second-level index.
5. The method according to any one of claims 1-3, characterized in that, Based on the vector knowledge base, the query dialogue is retrieved to determine multiple candidate intent tags, including: The query dialogue is preprocessed and transformed into a vector to obtain a dialogue vector. The preprocessing includes at least one of key content extraction and text normalization. Using a similarity algorithm, multiple candidate intent tags are retrieved from the vector knowledge base, and the similarity between the intent vector corresponding to the candidate intent tag and the dialogue vector is greater than or equal to a first threshold.
6. The method according to any one of claims 1-3, characterized in that, The method further includes: Stratified sampling evaluation is performed on the labeled information of multiple query dialogues within a preset time period to obtain feedback data; Based on the feedback data, the brief description or detailed description information corresponding to at least one intent tag in the vector knowledge base is updated.
7. The method according to claim 6, characterized in that, For any intent tag in the vector knowledge base, based on the feedback data, the brief description or detailed description information corresponding to the intent tag is updated, including: Based on the feedback data, determine whether the preset triggering condition corresponding to the intent tag is met; If so, determine the incorrectly labeled samples, correctly labeled samples, and feedback information corresponding to the intent label, and determine the reason for the labeling error corresponding to the intent label based on the incorrectly labeled samples, correctly labeled samples, and feedback information; Based on the cause of the labeling error, update description information is generated, which includes updated detailed description information and updated brief description information; The updated description information is subjected to effect verification processing, and after the verification is passed, the description information and intent vector of the corresponding intent tag in the vector knowledge base are updated according to the updated description information.
8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.