Data labeling method, device, equipment, medium and program product

Abstract

The application provides a data labeling method and device, equipment, medium and program product, comprising: obtaining first data; wherein the first data is associated with the implementation process of at least part of the function of an electronic device; pre-labeling the first data in at least two dimensions through a model in a plurality of models to obtain a set of labels; determining a target label from the set of labels based on the matching degree between the labels in the set of labels; and labeling the first data based on the target label. In the case of an electronic device being a vehicle, through the application, the automatic, efficient and accurate labeling of the first data associated with the implementation process of at least part of the function of the vehicle can be realized.

Description

Technical Field

[0001] This application relates to the field of vehicle data processing technology, and in particular to a data annotation method, apparatus, device, medium, and program product. Background Technology

[0002] In a vehicle data closed-loop scenario, capturing and identifying issues related to poor user experience is crucial for improving vehicle functionality and optimizing data processing performance. To address these technical challenges, a solution has been developed whereby professionals annotate user experience-related data in vehicle-uploaded logs and analyze the annotated data. However, the accuracy of this data annotation is insufficient. Summary of the Invention

[0003] Based on the above technical problems, embodiments of this application provide a data annotation method, apparatus, device, medium, and program product. When the electronic device is a vehicle, this solution can achieve automated, efficient, and accurate annotation of first data associated with the implementation process of at least some functions of the vehicle.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data annotation method, the method comprising:

[0006] Acquire first data; wherein the first data is associated with the implementation process of at least some functions of the electronic device;

[0007] The first data is pre-labeled using multiple models in at least two dimensions to obtain a set of identifiers.

[0008] Based on the degree of matching between the identifiers in the identifier set, the target identifier is determined from the identifier set;

[0009] The first data is labeled based on the target identifier.

[0010] In some embodiments, determining the target identifier from the identifier set based on the matching degree between identifiers in the identifier set includes:

[0011] If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the n-th identifier is combined with the first data to obtain the first combined data. The first combined data is then judged by the m-th model among the multiple models to obtain the first judgment result. If the first judgment result indicates that the n-th identifier is the correct identification for the first data, the n-th identifier is determined as the target identifier.

[0012] or,

[0013] If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the m-th identifier is combined with the first data to obtain second combined data. The second combined data is then judged by the n-th model among the multiple models to obtain a second judgment result. If the second judgment result indicates that the m-th identifier is a correct identification for the first data, the m-th identifier is determined as the target identifier.

[0014] Wherein, the m-th identifier is obtained by identifying the first data through the m-th model; the n-th identifier is obtained by identifying the first data through the n-th model; m and n are both integers greater than or equal to 1, and m and n are not equal.

[0015] In some embodiments, the method further includes:

[0016] If the first discrimination result indicates that the nth identifier is not a correct identification for the first data, or if the second discrimination result indicates that the mth identifier is not a correct identification for the first data, output a prompt message indicating that the multiple models cannot determine the identifier of the first data, so as to prompt the first data to be labeled through the specified labeling process.

[0017] In some embodiments, the first data is associated with a conversational interaction model of the electronic device; the at least partial function includes the conversational interaction function of the conversational interaction model; the identifier set includes emotion identifiers; the step of pre-labeling the first data in at least two dimensions using multiple models to obtain the identifier set includes:

[0018] First prompt data is constructed. Based on the first prompt data, the models among the multiple models are controlled to pre-label the conversation context data contained in the first data in the emotion dimension to obtain the emotion identifier.

[0019] In some embodiments, the first data is associated with a session interaction model of the electronic device; the at least partial function includes the session interaction function of the session interaction model; the identifier set includes preference identifiers; the step of pre-labeling the first data in at least two dimensions using models from multiple models to obtain the identifier set includes:

[0020] Construct second prompt data, and based on the second prompt data, control the models among the multiple models to pre-label the session context data contained in the first data in the dimension of user preference to obtain the preference identifier.

[0021] In some embodiments, after labeling the first data based on the target identifier, the method further includes:

[0022] Based on the first data labeled with the target identifier, construct a sample dataset;

[0023] An initial annotation model is trained based on the sample dataset to obtain the annotation model.

[0024] In some embodiments, the method further includes:

[0025] Obtain the second data to be labeled;

[0026] The second data is labeled in at least two dimensions using the labeling model.

[0027] This application embodiment also provides a data annotation device, the data annotation device comprising:

[0028] Acquire first data; wherein the first data is associated with the implementation process of at least some functions of the electronic device;

[0029] The first data is pre-labeled using multiple models in at least two dimensions to obtain a set of identifiers.

[0030] Based on the degree of matching between the identifiers in the identifier set, the target identifier is determined from the identifier set;

[0031] The first data is labeled based on the target identifier.

[0032] This application also provides an electronic device, which includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it can implement the data annotation method as described above.

[0033] This application also provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor of an electronic device, it can implement the data annotation method as described above.

[0034] This application also provides a computer program product, which includes a computer program; when the computer program is executed by the processor of an electronic device, it can implement the data annotation method as described above.

[0035] The data annotation method provided in this application, after obtaining first data associated with the implementation process of at least some functions of an electronic device, pre-annotates the first data in at least two dimensions using multiple models to obtain an identifier set. Thus, identifying the first data using multiple models reduces the probability of incorrect annotations caused by relying on a single model for annotation. Furthermore, pre-annotating the first data in at least two dimensions using models from multiple models improves the completeness and accuracy of the identifiers in the identifier set. Based on this, determining the target identifier from the identifier set based on the matching degree between identifiers in the identifier set reduces the probability of annotation errors caused by relying on a single model for pre-annotation of the first data, thereby improving the accuracy of the target identifier. Simultaneously, annotating the first data based on the target identifier enables automated, efficient, and accurate annotation of the first data.

[0036] In the case of vehicles as electronic devices, this solution enables automated, high-precision annotation of first data associated with the implementation process of at least some of the vehicle's functions. Attached Figure Description

[0037] Figure 1 A flowchart illustrating the data annotation method provided in this application embodiment;

[0038] Figure 2 A schematic diagram illustrating the process of training the initial labeled model provided in an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the structure for training the session data annotation and feedback evaluation model provided in an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the structure of the data annotation device provided in the embodiments of this application;

[0041] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0046] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] (1) Large Language Model (LLM): This is an artificial intelligence model based on deep learning technology, primarily used to understand and generate human language. The core of LLM is training on large amounts of text data to capture the complex relationships between words, phrases, and longer text segments. These models have a large number of parameters and are pre-trained on a large amount of unlabeled text data, thus achieving powerful language capabilities. Typical LLMs include MindGPT, ChatGPT, and Wenxin Yiyan, all of which can generate relevant responses based on user queries.

[0049] (2) Prompt: This refers to the text input into the LLM that guides it to generate specific outputs. A well-designed prompt has a significant impact on the performance of the LLM. It is often used in prompt engineering. Prompt engineering is about techniques for building prompts that can fully utilize the capabilities of large models for specific tasks. There are two key principles for designing efficient prompts: First, clear and specific instructions need to be written to trigger the LLM to more clearly and accurately capture and identify data processing needs; second, the LLM needs to be given sufficient time to think so that it can perform step-by-step inference within a sufficient time to obtain more accurate and reliable data processing results.

[0050] (3) Ensemble learning: This is a technique in machine learning that combines multiple different models to improve prediction or classification performance. The basic idea of ​​ensemble learning is that multiple models making decisions together are more accurate and robust than a single model.

[0051] (4) Few-Shot Learning: This involves providing the model with a small number of samples via a prompt, allowing it to learn from these samples and thus guide it to complete a specific task. For example, if English to French translation is required, several English-French examples can be pre-input into GPT-4, followed by English sentences. In this case, GPT-4 can generate the corresponding French translation.

[0052] In data loop scenarios involving the entire vehicle, applications, and service components, identifying or capturing the issues that most impact user experience and fixing them as quickly as possible becomes a data prerequisite for optimizing the entire vehicle, applications, or service components.

[0053] To identify the issues that most impact user experience, related technologies provide a pre-defined standard for right or wrong, or good or bad. Based on this standard, professional technicians annotate vehicle-related log data or rely on super models, including GPT, to score the data and determine whether the log data contains issues or defects that affect user experience.

[0054] However, in actual vehicle use, different vehicle users have different preferences, and the subjective nature of different users' feelings is relatively strong. Therefore, the same function execution process may not meet all the expectations of different users, and it is impossible to completely align the unique right or wrong standard or good or bad standard with the diverse and differentiated user preferences and subjectivity.

[0055] Secondly, in the process of manual annotation, on the one hand, it is impossible to eliminate the subjective differences between the professional technicians who perform the annotation operation and the vehicle users; on the other hand, the scalability of data labels annotated manually is relatively weak.

[0056] At the same time, scoring log data using supermodels relies entirely on the supermodel's own ability to follow rules and its own preferences. This makes it impossible for the annotation results to match differentiated business standards or to capture real human values ​​or subtle preferences with 100% accuracy.

[0057] To address the aforementioned technical issues, a solution has been developed whereby professional technicians annotate user experience-related data in vehicle-uploaded log data and then analyze the annotated data. However, the accuracy of this data annotation is insufficient.

[0058] Based on the above technical problems, embodiments of this application provide a data annotation method, apparatus, device, medium, and program product.

[0059] Figure 1 This is a flowchart illustrating the data annotation method provided in an embodiment of this application. This method can be implemented using a processor of an electronic device. Figure 1 As shown, the method may include the following steps:

[0060] S101, Obtain the first data.

[0061] The first data is associated with the implementation process of at least some functions of the electronic device.

[0062] In one embodiment, the electronic device may include a physical device or a virtual machine device, and may include a mobile electronic device or a non-mobile electronic device.

[0063] In one implementation, the electronic device may include a vehicle; for example, the vehicle may include a new energy vehicle or a smart vehicle powered by energy sources such as electricity or natural gas.

[0064] In one implementation, at least some of the functionality may be associated with the vehicle’s hardware and / or software; for example, the software may include system applications or user applications installed in the vehicle.

[0065] In one implementation, the first data may be used to record the implementation process of at least some of the vehicle's functions; for example, the first data may include vehicle log data.

[0066] In one implementation, the first data can be used to record the implementation process of at least some functions of multiple vehicles; for example, with the permission of the users of multiple vehicles, the first data can be sent to a cloud database associated with the vehicles; correspondingly, the first data can be retrieved from the cloud database.

[0067] In one implementation, the first data may correspond to a specified vehicle, model, time period, and component update status. For example, the first data may be associated with a specified model that is in the component update state and the implementation process of at least some functions within a specified time period.

[0068] S102. Pre-label the first data in at least two dimensions using multiple models to obtain a set of labels.

[0069] In one implementation, the multiple models may include a multi-model ensemble; for example, the models in the multiple models may be different from each other, but all of them can achieve the function of recognizing the first data.

[0070] In one implementation, all models in the plurality of models can be LLM models; for example, the plurality of models may include models such as GPT, Deepseek, and Qwen.

[0071] In one implementation, the identifiers in the identifier set can correspond to each of the multiple models. For example, the kth identifier in the identifier set can be an identifier obtained by identifying the first data through the kth model among the multiple models, and the number of identifiers in the kth identifier can be the same as the number of dimensions in at least two dimensions; where k can be an integer greater than or equal to 1.

[0072] In one implementation, the identifier set can be obtained in the following way:

[0073] For each of the multiple models, the first data is pre-labeled in at least two dimensions to obtain a set of identifiers; for example, the first data is identified in at least two dimensions by the k-th model in the multiple models to obtain the k-th identifier, and then the first identifier to the k-th identifier are combined to obtain a set of identifiers; where K is the number of models contained in the multiple models, and K is greater than or equal to 3, while k can be any integer greater than or equal to 1 and less than or equal to K.

[0074] In one implementation, at least two dimensions may include a function implementation dimension and a performance dimension; correspondingly, the identifiers in the identifier set may characterize user feedback on at least some functions in the function implementation dimension and the performance dimension; for example, the function implementation dimension may be used to characterize whether a function can be implemented, the completeness of the function implementation, and whether a certain function is available, etc.; for example, the performance dimension may include the speed of function implementation and / or the stability of function implementation, etc.

[0075] S103. Based on the degree of matching between identifiers in the identifier set, determine the target identifier from the identifier set.

[0076] In one implementation, the target identifier can be determined in the following way:

[0077] If the number of matching identifiers in the matching degree indicator set is greater than or equal to the first threshold, then the identifiers that are in the matching state and whose number is greater than or equal to the first threshold can be identified as target identifiers.

[0078] S104. Label the first data based on the target identifier.

[0079] In one implementation, the first data can be labeled in any of the following ways:

[0080] The first data is processed to obtain the valid data corresponding to the target identifier, and the target identifier is associated with the valid data, or a key-value pair consisting of the target data and the valid data is constructed, thereby achieving the annotation of the first data.

[0081] The target identifier is directly associated with the first data, thereby achieving the annotation of the first data.

[0082] As can be seen from the above, the data annotation method provided in this application, after obtaining first data associated with the implementation process of at least some functions of an electronic device, pre-annotates the first data in at least two dimensions using models from multiple models to obtain an identifier set. Thus, pre-annotating the first data using models from multiple models can reduce the probability of incorrect annotation caused by relying on a single model to identify the first data; furthermore, identifying the first data in at least two dimensions using models from multiple models can improve the completeness and accuracy of the identifiers in the identifier set; based on this, determining the target identifier from the identifier set based on the matching degree between the identifiers in the identifier set can reduce the probability of annotation errors caused by relying on a single model to pre-annotate the first data, thereby improving the accuracy of the target identifier; at the same time, annotating the first data based on the target identifier can achieve automated, efficient, and accurate annotation of the first data.

[0083] In the case of vehicles as electronic devices, this solution enables automated, high-precision annotation of first data associated with the implementation process of at least some of the vehicle's functions.

[0084] Based on the foregoing embodiments, in the data annotation method provided in this application, the target identifier is determined from the identifier set based on the matching degree between identifiers in the identifier set, and if the matching degree indicates that the number of identifiers in the identifier set that are in a matching state is greater than or equal to a first threshold, then the identifiers that are in a matching state and whose number is greater than or equal to the first threshold can be determined as the target identifier. This can all be achieved in the following ways:

[0085] If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the n-th identifier is combined with the first data to obtain the first combined data. The first combined data is then judged by the m-th model among multiple models to obtain the first judgment result. If the first judgment result indicates that the n-th identifier is the correct identification for the first data, the n-th identifier is determined as the target identifier.

[0086] or,

[0087] If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the m-th identifier is combined with the first data to obtain the second combined data. The second combined data is then judged by the n-th model among multiple models to obtain the second judgment result. If the second judgment result indicates that the m-th identifier is the correct identification for the first data, the m-th identifier is determined as the target identifier.

[0088] Wherein, the m-th identifier is obtained by identifying the first data through the m-th model, and the n-th identifier is obtained by identifying the first data through the n-th model; m and n are both integers greater than or equal to 1, and m and n are not equal.

[0089] Accordingly, if the m-th identifier matches the n-th identifier, then the operation of combining the n-th identifier with the first data or combining the m-th identifier with the first data can be omitted.

[0090] Accordingly, if the first discrimination result indicates that the nth identifier is not a correct identification for the first data, then the operation of identifying the nth identifier as the target identifier can be omitted.

[0091] Accordingly, if the second discrimination result indicates that the m-th identifier is not a correct identification for the first data, then the operation of identifying the m-th identifier as the target identifier can be omitted.

[0092] In one implementation, the first combined data may be represented in the form of key-value pairs or data combinations; for example, the first combined data may include first data and an nth identifier.

[0093] In one implementation, the second combined data may be represented in the form of key-value pairs or data combinations; for example, the second combined data may include the first data and the m-th identifier.

[0094] In one implementation, the first discrimination result indicates that the nth identifier is a correct identification for the first data, and may include the first discrimination result containing at least one of the following: the meaning, characteristics and type expressed by the first data, which matches the nth identifier.

[0095] Accordingly, the second discrimination result indicates that the first identifier is a correct identification for the first data, and may include the following: the second discrimination result contains at least one of the meaning, characteristics and type expressed by the first data, which matches the m-th identifier.

[0096] In one implementation, the first discrimination result can indicate whether the nth identifier is a correct identification for the first data.

[0097] In one implementation, the first combination of data is discriminated using the m-th model, which can be obtained in the following way:

[0098] A first prompt is constructed, and the first prompt and the first combination data are input into the m-th model to instruct the m-th model to judge whether the meaning expressed by the first data matches the n-th identifier, thereby obtaining a first discrimination result; wherein, the first prompt may include: judging whether the meaning expressed by the first data in the first combination data matches the n-th identifier.

[0099] In one implementation, the second discrimination result can indicate whether the m-th identifier is a correct identification for the first data.

[0100] In one implementation, the second combination of data is discriminated using the nth model, which can be obtained in the following way:

[0101] A second prompt is constructed, and the second prompt and the second combination data are input into the nth model to instruct the nth model to judge whether the meaning expressed by the first data matches the mth identifier, thereby obtaining a second discrimination result; wherein, the second prompt may include: judging whether the meaning expressed by the first data in the second combination data matches the mth identifier.

[0102] It should be noted that if the m-th identifier does not match the n-th identifier, it indicates that the first data is in the fuzzy zone between the m-th model and the n-th model in the data processing dimension, that is, the model among the multiple models cannot accurately identify and process the first data.

[0103] For example, when the m-th identifier and the n-th identifier do not match, a first combination of data is obtained, and the first combination of data is judged by the m-th model to obtain a first discrimination result. Then, based on the first discrimination result, it is determined whether the n-th identifier is identified as the target identifier. And, when the m-th identifier and the n-th identifier do not match, a second combination of data is obtained, and the second combination of data is judged by the n-th model to obtain a second discrimination result. Then, based on the second discrimination result, it is determined whether the m-th identifier is identified as the target identifier. This realizes the cross discrimination of identifiers in the identifier set. Therefore, the above process can be referred to as the cross discrimination process.

[0104] As can be seen from the above, the data annotation method provided in this application embodiment, if the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, then the n-th identifier is combined with the first data to obtain the first combined data, and the first combined data is judged by the m-th model among multiple models to obtain the first judgment result. If the first judgment result indicates that the n-th identifier is the correct identification for the first data, then the n-th identifier is determined as the target identifier. Alternatively, if the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the m-th identifier is combined with the first data to obtain the second combined data, and the second combined data is judged by the n-th model among multiple models to obtain the second judgment result. If the second judgment result indicates that the m-th identifier is the correct identification for the first data, then the m-th identifier is determined as the target identifier. The m-th identifier and the n-th identifier are respectively obtained by identifying the first data through the m-th model and the n-th model. Thus, through the above operations, the cross-discrimination of the nth and mth identifiers by the mth and nth models respectively is achieved. This not only leverages the advantages of different models in the data recognition dimension, but also reduces the probability of misjudging the target identifier.

[0105] Based on the foregoing embodiments, the data annotation method provided in this application can also perform the following operations:

[0106] If the first discrimination result indicates that the nth identifier is not a correct identification for the first data, or if the second discrimination result indicates that the mth identifier is not a correct identification for the first data, output a prompt message that multiple models cannot determine the identifier of the first data, so as to prompt the first data to be labeled through the specified labeling process.

[0107] Accordingly, if the first discrimination result indicates that the nth identifier is the correct identification for the first data, or if the second discrimination result indicates that the mth identifier is the correct identification for the first data, then the nth identifier can be determined as the target identifier, or the mth identifier can be determined as the target identifier, by means of the method provided in the foregoing embodiments.

[0108] In one implementation, if the first discrimination result indicates that the nth identifier is not a correct identification for the first data, or if the second discrimination result indicates that the mth identifier is not a correct identification for the first data, a prompt message indicating that multiple models cannot determine the identifier of the first data is output. This can indicate that the first data has exceeded the data recognition range of the mth model and the nth model, or that the first data has exceeded the common or widely adopted data processing range in the field of vehicle technology, but covers a specific field or specific scenario of the vehicle.

[0109] In one implementation, the prompt message can be presented in the form of text.

[0110] In one implementation, specifying the annotation process may include manually annotating the first data by professional technicians.

[0111] As can be seen from the above, the data annotation method provided in this application, if the first discrimination result indicates that the nth identifier is not a correct identification for the first data, or if the second discrimination result indicates that the mth identifier is not a correct identification for the first data, outputs a prompt message indicating that multiple models cannot determine the identifier of the first data, so as to prompt the first data to be annotated through a specified annotation process. In this way, when a model among multiple models cannot identify the first data, a prompt message can be output in a timely manner, thereby realizing a real-time reminder that a model among multiple models cannot accurately identify the first data; and, annotating the first data through a specified annotation process can also improve the flexibility and robustness of the first data annotation, thereby meeting the diverse annotation needs of the first data.

[0112] Based on the foregoing embodiments, in the data annotation method provided in this application, the first data is associated with the conversational interaction model of the electronic device; at least some functions include the conversational interaction function of the conversational interaction model; and the identifier set includes emotion identifiers.

[0113] In one implementation, when the electronic device is a vehicle, the conversational interaction model can enable conversational interaction between the vehicle and at least one vehicle occupant; for example, the conversational interaction model can have voice recognition, voiceprint recognition, data retrieval, and audio data output functions; for example, the conversational interaction model can rely on LLM to implement its conversational interaction functions.

[0114] In one implementation, the session interaction module can track, capture, and identify query requests sent by vehicle occupants through the session interaction function, identify the query requests, perform data retrieval operations, and then output response data.

[0115] Specifically, the query request can be a Query, and the response data can be a Response. The Query can be a question or instruction submitted by the vehicle occupant to the conversational interaction model, such as "Where is the capital of China?" or "Give me a brief introduction to the history of Nanjing". The Response is used to represent the reply content generated by the conversational interaction model based on the user's Query.

[0116] In one implementation, the emotion identifier can be used to characterize the emotions contained in the first data; for example, the emotions may include satisfaction and / or dissatisfaction.

[0117] Accordingly, by pre-labeling the first data in at least two dimensions using models from multiple models, a set of labels can be obtained, which can be achieved in the following way:

[0118] First prompt data is constructed. Based on the first prompt data, multiple models are controlled to pre-label the conversation context data contained in the first data in the emotion dimension to obtain emotion labels.

[0119] In one implementation, the session context data may consist of multiple sessions comprising a Query and a Response from the first data; for example, a Query and a Response may constitute a session.

[0120] In one implementation, the first prompt data can be presented in the form of a prompt; for example, in order to improve the efficiency of the models in multiple models in recognizing the session context data, the first prompt data can be set in the form of key-value pairs.

[0121] In one implementation, emotion labels can be obtained in the following way:

[0122] The first prompt data and the first data are input into the k-th model among multiple models. The k-th model parses the sessions in the first data sequentially according to the time order based on the first prompt data to obtain the first session set. Then, it performs emotion recognition on each query in the first session set to obtain an emotion label. For example, the k-th model can generate a rating including three dimensions: satisfaction, dissatisfaction, and dialogue status for any query. The ratings of the above three dimensions are combined and output as emotion labels in the form of key-value pairs. It can also output the reasons for each of the above ratings. For example, the key-value pairs can be JavaScript Object Notation (JSON). The dialogue status can include state data such as the number of rounds of dialogue between the user and the session interaction model and the correlation between the queries sent by the user. The satisfaction dimension rating can represent the user's satisfaction with the session interaction function, and the dissatisfaction dimension rating can represent the user's dissatisfaction with the session interaction function.

[0123] It should be noted that, for the vehicle's conversational interaction model, a Query received by the model and its corresponding Response constitute a Session. Furthermore, each Session can correspond to a single round of Query and Response. In practical applications, it's possible to pre-analyze whether the queries and responses from multiple rounds contain emotion-related data. If they do, the emotion identifier of the first data represented by the queries and responses from those rounds can be determined.

[0124] For example, if the m-th emotion label output by the m-th model matches the n-th emotion label output by the n-th model, then the m-th emotion label or the n-th emotion label can be set as the final target emotion label, and the target label can include the target emotion label; wherein, the matching of the m-th emotion label and the n-th emotion label can include the same score corresponding to the m-th emotion label and the same score corresponding to the n-th emotion label, and can also include the same text corresponding to the m-th emotion label and the same text corresponding to the n-th emotion label.

[0125] For example, during the process of identifying the conversation context data contained in the first data, if the m-th emotion identifier does not match the n-th emotion identifier, the cross-discrimination process provided in the aforementioned embodiments can be used to process the m-th emotion identifier, the n-th emotion identifier, and the conversation context data in the first data. For instance, the n-th model can be used to discriminate the m-th emotion identifier and the conversation context data in the first data to obtain a first emotion discrimination result, or the m-th model can be used to discriminate the n-th emotion identifier and the conversation context data in the first data to obtain a second emotion discrimination result. The final target emotion identifier is then determined based on the first or second emotion discrimination result. The mismatch between the m-th and n-th emotion identifiers may include differences between the ratings corresponding to the m-th and n-th emotion identifiers, or inconsistencies between the text corresponding to the m-th and n-th emotion identifiers.

[0126] For example, if the target emotion identifier cannot be determined based on the first emotion identification result or the second emotion identification result, a prompt message can be output to trigger professional technicians to identify and label the emotion dimension of the conversation context contained in the first data.

[0127] As can be seen from the above, the data annotation method provided in this application embodiment associates the first data with the conversational interaction module of the electronic device, and at least some of its functions include the conversational interaction function of the conversational interaction model. After constructing the first prompt data, based on the first prompt data, the models in multiple models are controlled to identify the conversational context data contained in the first data in the emotion dimension to obtain an emotion label. In this way, through the above process, accurate identification of the conversational interaction function of the electronic device's conversational interaction model in the emotion dimension can be achieved; and by constructing the first prompt data, the accuracy and relevance of the models in multiple models in identifying the conversational context contained in the first data can be improved, thereby improving the accuracy of the emotion label.

[0128] Based on the foregoing embodiments, in the data annotation method provided in this application, the first data is associated with the session interaction model of the electronic device; at least some functions include the session interaction function of the session interaction model; and the identifier set includes preference identifiers.

[0129] In one implementation, the preference identifier can indicate the vehicle user's personal preferences, such as expecting the conversational interaction function to provide a more concise and richer response, a shorter response time, and the inclusion of options in the response.

[0130] Accordingly, by pre-labeling the first data in at least two dimensions using models from multiple models, a set of labels can be obtained, which can also be achieved in the following ways:

[0131] Construct a second set of prompt data; based on the second set of prompt data, control the models in multiple models to pre-label the session context data contained in the first set of prompt data in the dimension of user preferences, and obtain preference identifiers.

[0132] In one implementation, the second prompt data can be presented in the form of a prompt; for example, in order to improve the efficiency of pre-annotation of session context data by models in multiple models, the second prompt data can be set in the form of key-value pairs.

[0133] In one implementation, the preference identifier can be obtained in the following way:

[0134] The second prompt data and the first data are input into the k-th model among multiple models, so that the k-th model can parse the sessions in the first data sequentially from the time order based on the second prompt data to obtain the second session set. Then, data extraction of the user preference dimension is performed on each query in the second session set to obtain the preference identifier. For example, the k-th model can also output the reason for determining the preference identifier.

[0135] For example, if the m-th preference identifier output by the m-th model matches the n-th preference identifier output by the n-th model, then either the m-th preference identifier or the n-th preference identifier can be set as the final target preference identifier, and the target identifier can include the target preference identifier. Matching the m-th preference identifier with the n-th preference identifier can include the score corresponding to the m-th preference identifier being the same as the score corresponding to the n-th preference identifier, or it can include the text corresponding to the m-th preference identifier being identical to the text corresponding to the n-th preference identifier.

[0136] For example, during the process of identifying the conversation context data contained in the first data, if the m-th preference identifier and the n-th preference identifier do not match, the cross-discrimination process provided in the aforementioned embodiments can be used to process the m-th sentiment identifier, the n-th sentiment identifier, and the conversation context data in the first data. For instance, the n-th model can be used to discriminate the m-th preference identifier and the conversation context data in the first data to obtain a first preference discrimination result, or the m-th model can be used to discriminate the n-th preference identifier and the conversation context data in the first data to obtain a second preference discrimination result. The final preference identifier is then determined based on either the first or second preference discrimination result. The mismatch between the m-th and n-th preference identifiers may include differences between the scores corresponding to the m-th and n-th preference identifiers, or inconsistencies between the text corresponding to the m-th and n-th preference identifiers.

[0137] For example, if the target emotion identifier cannot be determined based on the first preference discrimination result or the second preference discrimination result, a prompt message can be output to trigger professional technicians to identify and label the user preference dimension of the conversation context contained in the first data.

[0138] For example, after obtaining the first data corresponding to the conversation interaction module, the conversation context data contained in the first data can be analyzed by the models in multiple models respectively, so as to determine multiple emotion identifiers and multiple preference identifiers respectively. At this time, the identifier set may include multiple emotion identifiers and multiple preference identifiers.

[0139] For example, if there is a match between corresponding types of identifiers in the identifier set, the matching identifiers can be identified as identifiers of that type in the target identifier set. For instance, if there is a match between multiple emotion identifiers, any one of the multiple emotion identifiers can be identified as the target emotion identifier.

[0140] If the corresponding types of identifiers do not match, such as the corresponding types of identifiers having different ratings or inconsistent texts, the mismatched corresponding types of identifiers can be processed through the cross-discrimination process provided in the aforementioned embodiments to determine the final identifier of the corresponding type, such as the target emotion identifier and the target preference identifier. The final target identifier may include the target emotion identifier and the target preference identifier.

[0141] As can be seen from the above, in the data annotation method provided in this application embodiment, the first data is associated with the conversational interaction model of the electronic device, and at least some of its functions include the conversational interaction functions of the conversational interaction model. The identifier set includes preference identifiers. Furthermore, based on the second prompt data, multiple models are constructed and controlled to identify the conversational context data contained in the first data in the user preference dimension to obtain preference identifiers. Thus, through the above process, accurate identification of the conversational interaction functions of the electronic device's conversational interaction model in the user preference dimension can be achieved. Moreover, by constructing the second prompt data, the accuracy and relevance of the identification of the conversational context contained in the first data by the multiple models can be improved, thereby improving the accuracy of preference identifiers.

[0142] Figure 2 This is a schematic diagram of the process for training the initial labeled model provided in an embodiment of this application, as shown below. Figure 2 As shown, the process may include the following steps:

[0143] S201, Obtain the first data.

[0144] The first data is associated with the implementation process of at least some functions of the electronic device.

[0145] S202. Pre-label the first data in at least two dimensions using models from multiple models to obtain a set of labels.

[0146] S203. Based on the degree of matching between identifiers in the identifier set, determine the target identifier from the identifier set.

[0147] S204. First data is labeled based on the target identifier.

[0148] S205. Construct a sample dataset based on the first data after labeling the target.

[0149] In one implementation, the sample dataset may include standardized unambiguous data, ambiguous data at the edges of the recognition range of multiple models, and customized business data in the field of vehicle technology.

[0150] The standardized and unambiguous data may include descriptive data corresponding to the implementation process of common or general standards in the field of vehicle technology. For example, by using the m-th model and the n-th model in multiple models to obtain the m-th identifier and the n-th identifier respectively for the first data, and the m-th identifier matches the n-th identifier, the first data can be standardized and unambiguous data.

[0151] The ambiguous data at the edge of the recognition range of multiple models may correspond to the mismatch between the m-th identifier and the n-th identifier in the aforementioned embodiments. However, by constructing the first combination data or the second combination data, the first data of the target identifier can be finally determined.

[0152] Customized business data in the field of vehicle technology may include first data where the target identifier cannot be determined through the technical solutions provided in the aforementioned embodiments, but is determined through manual annotation.

[0153] Through the above operations, semi-automatic annotation, collection, and expansion of the sample dataset are achieved. Furthermore, by using diverse and flexible annotation methods for the first data in the sample dataset, not only can the accuracy of the annotation of the first data be improved, but also the diversity and breadth of the sample data in the sample dataset can be expanded, thereby enabling the sample dataset to cover a variety of scenarios and standard ranges in the field of vehicle technology.

[0154] S206. Train the initial annotation model based on the sample dataset to obtain the annotation model.

[0155] In one implementation, the annotation model may be a model specifically designed to annotate data associated with at least some of the vehicle's functions; for example, the annotation model may determine the sentiment and preference labels for the conversation context data corresponding to the vehicle's conversational interaction model.

[0156] In one implementation, the initial annotation model may include an LLM or Artificial Intelligence (AI) model, which may have been pre-tuned and has the ability to identify log data corresponding to at least some of the vehicle's functions.

[0157] In one implementation, the annotation model can be obtained in the following way:

[0158] The first data is input into the initial annotation model to identify the first data and determine its initial identifier. Then, based on the difference between the initial identifier and the target identifier, at least some parameters of the initial annotation model are adjusted to obtain the initial annotation model after the first parameter adjustment. The first data is then input into the initial annotation model after the xth parameter adjustment to obtain the xth identifier. Based on the difference between the xth identifier and the target identifier, at least some parameters of the initial annotation model after the xth parameter adjustment are adjusted to obtain the initial annotation model after the (x+1)th parameter adjustment. The above parameter adjustment process is recursively executed until the difference between the identifier output by the initial annotation model after parameter adjustment and the target identifier is less than or equal to a second threshold. At this point, the initial annotation model after parameter adjustment can be determined as the annotation model. Here, x can be an integer greater than or equal to 1.

[0159] By leveraging the concept of few-shot learning, the sample dataset is input into the initial annotation model to trigger the initial annotation model to capture and learn the inherent relationship between the first data in the sample dataset and the target label, thereby obtaining the annotation model.

[0160] As can be seen from the above, the data annotation method provided in this application, based on the first data after the target identifier is labeled, constructs a sample dataset, and then trains an initial annotation model based on the sample dataset to obtain the annotation model. Thus, through the above operations, the first data after the target identifier is labeled is fully utilized, and targeted training of the initial annotation model is also achieved. Furthermore, the annotation model trained by the above method can accurately identify and annotate function-related data covering diverse vehicle business scenarios.

[0161] Based on the foregoing embodiments, the data annotation method provided in this application can also perform the following operations after obtaining the annotation model:

[0162] Obtain the second data to be labeled, and label the second data in at least two dimensions using the labeling model.

[0163] In one implementation, the scenarios and functional scope covered by the second data may be the same as those covered by the first data.

[0164] In one implementation, the second data may include the software and / or hardware-related functional implementation process of the vehicle. For example, the second data may cover the implementation process of the human-machine interaction function of the vehicle's human-machine interaction model.

[0165] In one implementation, the second data can be input into the annotation model to obtain the data identifier corresponding to the second data, and the second data can be annotated based on the data identifier.

[0166] As can be seen from the above, the data annotation method provided in this application, after obtaining the second data to be annotated, annotates the second data in at least two dimensions through the annotation model, which can achieve efficient and accurate annotation of the second data.

[0167] Figure 3 This is a schematic diagram illustrating the structure of the session data annotation and feedback evaluation model training provided in an embodiment of this application. Figure 3 As shown, log data of the vehicle's conversational interaction model during the execution of conversational interaction functions can be uploaded through the real vehicle / APP online environment. Conversational data can be obtained by preprocessing the log data, which can be the conversational context data in the aforementioned embodiment. For example, after obtaining the log data, initial conversational data can be obtained from the log data, and preprocessing can be performed on it to obtain conversational context data. For example, preprocessing can include removing dirty data such as data with excessively short sessions, empty data, and abnormal data from the initial conversational data.

[0168] For example, after obtaining the conversation context data, feedback signal recognition can be performed on the conversation context data through multiple models included in the ensemble to obtain emotion identifiers. Feedback tendency processing can also be performed on the conversation data through models in the multiple models to obtain preference identifiers. For example, the multiple models can include the first model to the third model, and can also include more models.

[0169] For example, if the identifiers in the identifier sets output by multiple models are completely identical, that is, the identifiers in the identifier sets match each other, then the target identifier can be determined as Data1.

[0170] For example, if the identifiers in the identifier sets output by multiple models do not match, the cross-discrimination process provided in the foregoing embodiments can be used to perform cross-discrimination and cross-validation on the first combination data or the second combination data, thereby determining the identifier Data2, which is determined to be the correct identifier for the session data, as the target identifier.

[0171] For example, if the identifier of the session data still cannot be determined through the cross-discrimination process, the identifier of the session data, namely Data3, can be determined by manual annotation, and the target identifier may include Data3.

[0172] For example, by determining Data1 to Data3 through the preceding steps, diverse, flexible and accurate annotation of the session data is achieved. At this point, the annotated session data can be determined as session sample data, and training data augmentation and model tuning can be performed based on the session sample data.

[0173] For example, the labeled session sample data can be added to the existing session sample set to expand the session sample set.

[0174] For example, the parameters of the feedback evaluation model can be tuned or trained based on the expanded session sample set, thereby improving the accuracy of the data annotation of the feedback evaluation model; for example, the feedback evaluation model can be the annotation model in the foregoing embodiments.

[0175] For example, after the feedback evaluation model is tuned, feedback signal identification and / or feedback tendency extraction can be performed on the subsequently received conversation data to determine the sentiment identifier and / or preference identifier of the subsequently received conversation data, and the subsequently received conversation data can be labeled.

[0176] Through the above process, the identification of session data and the optimization of the feedback evaluation model can be integrated, thereby improving the targeting of the training of the feedback evaluation model and enhancing the accuracy and efficiency of the feedback evaluation model in labeling subsequent session data. The feedback evaluation model can be the labeling model described in the preceding embodiments.

[0177] This application also provides a data annotation device. Figure 4 This is a schematic diagram of the structure of the data annotation device provided in the embodiments of this application, such as... Figure 4 As shown, the data annotation device 4 includes:

[0178] The acquisition module 401 is used to acquire first data; wherein the first data is associated with the implementation process of at least some functions of the electronic device;

[0179] The identification module 402 is used to pre-label the first data in at least two dimensions using models from multiple models to obtain an identifier set;

[0180] The annotation module 403 is used to determine the target identifier from the identifier set based on the degree of matching between identifiers in the identifier set; and to annotate the first data based on the target identifier.

[0181] In some embodiments, the annotation module 403 is configured to, if the mth identifier in the identifier set does not match the nth identifier in the identifier set, combine the nth identifier with the first data to obtain first combined data, and use the mth model among multiple models to judge the first combined data to obtain a first judgment result; if the first judgment result indicates that the nth identifier is the correct identification for the first data, determine the nth identifier as the target identifier.

[0182] or,

[0183] If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the m-th identifier is combined with the first data to obtain the second combined data. The second combined data is then judged by the n-th model among multiple models to obtain the second judgment result. If the second judgment result indicates that the m-th identifier is the correct identification for the first data, the m-th identifier is determined as the target identifier.

[0184] Wherein, the m-th identifier is obtained by identifying the first data through the m-th model; the n-th identifier is obtained by identifying the first data through the n-th model; m and n are both integers greater than or equal to 1, and m and n are not equal.

[0185] In some embodiments, the annotation module 403 is configured to output a prompt message indicating that multiple models cannot determine the identifier of the first data if the first discrimination result indicates that the nth identifier is not a correct identification for the first data, or if the second discrimination result indicates that the mth identifier is not a correct identification for the first data, so as to prompt the first data to be annotated through a specified annotation process.

[0186] In some embodiments, the first data is associated with a conversational interaction model of an electronic device; at least some of the functions include conversational interaction functions of the conversational interaction model; the identifier set includes emotion identifiers;

[0187] The recognition module 402 is used to construct the first prompt data. Based on the first prompt data, it controls the models in multiple models to pre-label the conversation context data contained in the first data in the emotion dimension to obtain the emotion label.

[0188] In some embodiments, the first data is associated with a session interaction model of an electronic device; at least some of the functions include the session interaction functions of the session interaction model; the identifier set includes preference identifiers;

[0189] The identification module 402 is used to construct second prompt data. Based on the second prompt data, it controls the models in multiple models to pre-label the session context data contained in the first data in the user preference dimension to obtain preference identifiers.

[0190] In some embodiments, the data annotation model further includes a training module for constructing a sample dataset based on the first data after the annotation target is identified; and training an initial annotation model based on the sample dataset to obtain the annotation model.

[0191] In some embodiments, the acquisition module 401 is used to acquire the second data to be labeled;

[0192] The annotation module 403 annotates the second data in at least two dimensions using an annotation model.

[0193] This application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device includes a processor 501 and a memory 502; the memory stores a computer program; when the computer program is executed by the processor, it can implement the data annotation method as described above.

[0194] This application also provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor of an electronic device, it can implement the data annotation method as described above.

[0195] This application also provides a computer program product, which includes a computer program; when the computer program is executed by the processor of an electronic device, it can implement the data annotation method as described above.

[0196] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0197] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0198] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

Claims

1. A data annotation method, characterized in that, The method includes: Acquire first data; wherein the first data is associated with the implementation process of at least some functions of the electronic device; The first data is pre-labeled using multiple models in at least two dimensions to obtain a set of identifiers. Based on the degree of matching between the identifiers in the identifier set, the target identifier is determined from the identifier set; The first data is labeled based on the target identifier.

2. The method according to claim 1, characterized in that, The step of determining the target identifier from the identifier set based on the matching degree between identifiers in the identifier set includes: If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the n-th identifier is combined with the first data to obtain the first combined data. The first combined data is then judged by the m-th model among the multiple models to obtain the first judgment result. If the first judgment result indicates that the n-th identifier is the correct identification for the first data, the n-th identifier is determined as the target identifier. or, If the m-th identifier in the identifier set does not match the n-th identifier in the identifier set, the m-th identifier is combined with the first data to obtain second combined data. The second combined data is then judged by the n-th model among the multiple models to obtain a second judgment result. If the second judgment result indicates that the m-th identifier is a correct identification for the first data, the m-th identifier is determined as the target identifier. Wherein, the m-th identifier is obtained by identifying the first data through the m-th model; the n-th identifier is obtained by identifying the first data through the n-th model; m and n are both integers greater than or equal to 1, and m and n are not equal.

3. The method according to claim 2, characterized in that, The method further includes: If the first discrimination result indicates that the nth identifier is not a correct identification for the first data, or if the second discrimination result indicates that the mth identifier is not a correct identification for the first data, output a prompt message indicating that the multiple models cannot determine the identifier of the first data, so as to prompt the first data to be labeled through the specified labeling process.

4. The method according to any one of claims 1 to 3, characterized in that, The first data is associated with the conversational interaction model of the electronic device; the at least some functions include the conversational interaction functions of the conversational interaction model; the identifier set includes emotion identifiers; The step involves pre-labeling the first data using multiple models in at least two dimensions to obtain a set of identifiers, including: First prompt data is constructed. Based on the first prompt data, the models among the multiple models are controlled to pre-label the conversation context data contained in the first data in the emotion dimension to obtain the emotion identifier.

5. The method according to any one of claims 1 to 4, characterized in that, The first data is associated with the session interaction model of the electronic device; the at least some functions include the session interaction functions of the session interaction model; the identifier set includes preference identifiers; The step involves pre-labeling the first data using multiple models in at least two dimensions to obtain a set of identifiers, including: Construct second prompt data, and based on the second prompt data, control the models among the multiple models to pre-label the session context data contained in the first data in the dimension of user preference to obtain the preference identifier.

6. The method according to any one of claims 1 to 5, characterized in that, After labeling the first data based on the target identifier, the process further includes: Based on the first data labeled with the target identifier, construct a sample dataset; An initial annotation model is trained based on the sample dataset to obtain the annotation model.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the second data to be labeled; The second data is labeled in at least two dimensions using the labeling model.

8. A data annotation device, characterized in that, The data annotation device includes: An acquisition module is used to acquire first data; wherein the first data is associated with the implementation process of at least some functions of the electronic device; The identification module is used to pre-label the first data in at least two dimensions using multiple models to obtain an identifier set; The annotation module is used to determine a target identifier from the identifier set based on the degree of matching between identifiers in the identifier set; and to annotate the first data based on the target identifier.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it can implement the data annotation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed by the processor of the electronic device, it is able to implement the data annotation method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The program product includes a computer program; when the computer program is executed by the processor of an electronic device, it is capable of implementing the data annotation method as described in any one of claims 1 to 7.

Images