Intelligent marking model training method and device and intelligent marking method and device

Abstract

The invention provides an intelligent marking model training method and device and an intelligent marking method and device, relates to the artificial intelligence field of computer vision, deep learning, large models and the like, and can be applied to scenes of intelligent interaction and the like. The intelligent marking model training method comprises the steps that a first data set composed of first training data is acquired, wherein the first training data comprises dialogue information and answers corresponding to the dialogue information; generating user requirements corresponding to the first training data, and constructing second training data according to the first training data and the corresponding user requirements; and forming a second data set by using the constructed second training data, and training an intelligent marking model by using the second data set, the intelligent marking model being used for generating a user demand corresponding to third training data, the third training data comprising dialogue information and an answer corresponding to the dialogue information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of computer vision, deep learning and large models, and specifically to intelligent labeling model training and intelligent labeling methods and devices. Background Art

[0002] With the development of deep learning technology and the reduction of computing power costs, large models have been used more and more widely. For example, large models can be used for dialogue, that is, to generate corresponding answers for the dialogue information input by the user. Large models can include large language models (LLM, Large Language Model) and multimodal large models (MLLM, Multimodal LargeLanguage Model). In order to meet the needs of users, developers often need to use massive training data to train large models. These training data contain user needs (that is, user intentions). Understanding the distribution of user needs can help developers better understand the advantages and disadvantages of training data to configure the appropriate data ratio, and can help developers understand the characteristics of the data to analyze the reasons for differences in model performance in various occasions. Summary of the invention

[0003] The present invention provides intelligent labeling model training and intelligent labeling method and device.

[0004] A smart labeling model training method, comprising:

[0005] Acquire a first data set consisting of first training data, where the first training data includes dialogue information and answers corresponding to the dialogue information;

[0006] Generating user requirements corresponding to each first training data respectively, and constructing second training data according to the first training data and the corresponding user requirements;

[0007] The constructed second training data is used to form a second data set, and the second data set is used to train an intelligent labeling model, wherein the intelligent labeling model is used to generate user requirements corresponding to third training data, wherein the third training data includes dialogue information and answers corresponding to the dialogue information.

[0008] An intelligent marking method, comprising:

[0009] Acquire third training data, wherein the third training data includes dialogue information and answers corresponding to the dialogue information;

[0010] The user needs corresponding to the third training data are generated using an intelligent labeling model, wherein the intelligent labeling model is trained using the second training data in the second data set, and the second training data are constructed based on the first training data in the first data set and the user needs corresponding to the first training data, respectively, and the first training data include dialogue information and answers corresponding to the dialogue information.

[0011] An intelligent marking model training device comprises: a first processing module, a second processing module and a model training module;

[0012] The first processing module is used to obtain a first data set consisting of first training data, where the first training data includes dialogue information and answers corresponding to the dialogue information;

[0013] The second processing module is used to generate user requirements corresponding to each first training data, and construct second training data according to the first training data and the corresponding user requirements;

[0014] The model training module is used to use the constructed second training data to form a second data set, and use the second data set to train the intelligent labeling model, the intelligent labeling model is used to generate user needs corresponding to the third training data, and the third training data includes dialogue information and answers corresponding to the dialogue information.

[0015] A marking device, comprising: a data acquisition module and a marking processing module;

[0016] The data acquisition module is used to acquire third training data, wherein the third training data includes dialogue information and answers corresponding to the dialogue information;

[0017] The labeling processing module is used to generate user needs corresponding to the third training data using an intelligent labeling model, wherein the intelligent labeling model is trained using each second training data in the second data set, and each second training data is constructed based on each first training data in the first data set and the user needs corresponding to each first training data, and the first training data includes dialogue information and answers corresponding to the dialogue information.

[0018] An electronic device, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0022] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0023] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 This is a flow chart of an embodiment of the intelligent labeling model training method disclosed in the present invention;

[0027] Figure 2 It is a flow chart of an embodiment of the intelligent marking method disclosed in the present invention;

[0028] Figure 3 It is a schematic diagram of the overall implementation process of the intelligent marking method disclosed in the present invention;

[0029] Figure 4 It is a schematic diagram of the composition structure of an embodiment 400 of the intelligent marking model training device described in the present disclosure;

[0030] Figure 5 It is a schematic diagram of the structure of the marking device embodiment 500 of the present disclosure;

[0031] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0034] Figure 1 Flow chart of an embodiment of the intelligent labeling model training method described in the present disclosure. Figure 1 As shown, the following specific implementation methods are included.

[0035] In step 101, a first data set consisting of first training data is obtained, where the first training data includes dialogue information and answers corresponding to the dialogue information.

[0036] In step 102, user requirements corresponding to each first training data are generated respectively, and second training data are constructed according to the first training data and the corresponding user requirements.

[0037] In step 103, the constructed second training data is used to form a second data set, and the second data set is used to train the intelligent labeling model, the intelligent labeling model is used to generate user needs corresponding to the third training data, and the third training data includes dialogue information and answers corresponding to the dialogue information.

[0038] There is currently no better way to obtain user needs corresponding to training data.

[0039] By adopting the scheme described in the above method embodiment, a second data set can be constructed based on a first data set composed of first training data, and then an intelligent labeling model can be trained based on the second training data in the second data set. The intelligent labeling model can then be used to perform demand labeling on the third training data, that is, to generate user demands corresponding to the third training data. The third training data can be the training data in the third data set used to train a certain large model. Accordingly, the developers of the large model can have a better understanding of the distribution of user demands in the third data set, so as to facilitate the developers to adjust and optimize the third data set.

[0040] The intelligent labeling model can be a large language model or a multimodal large language model. In addition, the dialogue information input by the user may include: questions, or questions and pictures corresponding to the questions. In other words, the dialogue information usually includes questions, but whether to include pictures depends on actual needs. For example, the user gives a picture of a scenery and asks the following question: Please help me write a poem to describe the beautiful scenery in the picture. In this case, a picture is required. For another example, the question asked by the user is: In which year was the ** country established? In this case, no picture is required.

[0041] In addition, in the solution described in the present disclosure, a domain list and a behavior list can be pre-generated. Most of the requirements in the open scene can be described by a combination of "one noun + one verb", such as "text recognition", "animal recognition", "poetry writing", "chemistry problem solving", etc. The solution described in the present disclosure also follows the combination of "one noun + one verb" and abstracts user requirements into "domain + behavior".

[0042] For example, behaviors can be divided into the following categories: identification, question and answer, analysis, translation, problem solving, and creation. There may be overlaps between these behaviors, and a user demand may include multiple fields and multiple behaviors at the same time. The scope of the field is more complicated, mainly manifested in the serious inclusion relationship between the field itself, such as the inclusion relationship between "plants" and "roses", the inclusion relationship between "mathematics" and "statistics", etc. Ignoring this inclusion relationship may make the field division results confusing and have no reference value. Since knowledge graphs naturally have a cascading relationship, developers who have relevant information reserves can directly refer to the knowledge graph to divide the field, while developers who do not have relevant information reserves can divide the field through the directory structure of various encyclopedia websites and reference websites.

[0043] Accordingly, a domain list and a behavior list may be generated respectively. Furthermore, a segmented scene mapping table may be generated in advance.

[0044] That is, for some specific fields and / or behaviors, the requirements of segmented scenarios can be further constrained. For example, for the behavior of "translation", the intelligent labeling model can be required to further output segmented scenario requirements such as "source language" and "target language".

[0045] The segmented scene mapping table may include a first segmented scene mapping table and a second segmented scene mapping table, wherein the first segmented scene mapping table may include: some fields selected from the field list and the segmented requirements corresponding to each field, and the second segmented scene mapping table may include: some behaviors selected from the behavior list and the segmented requirements corresponding to each behavior. The segmented requirements may refer to determining two segmented scene requirements of "source language" and "target language", etc.

[0046] After the above processing is completed, user requirements corresponding to each first training data in the first data set may be generated respectively.

[0047] If manual labeling is used directly, that is, manually labeling the user needs corresponding to each first training data, it will take a lot of manpower and time costs, and the requirements for the knowledge reserve of the labeling personnel are extremely high, which is basically impossible to achieve.

[0048] Accordingly, in some embodiments of the present disclosure, cascade labeling results corresponding to each first training data can be generated respectively, each cascade labeling result includes at least two cascaded demand labels, and then the user needs corresponding to each first training data can be determined respectively according to the corresponding cascade labeling results.

[0049] It can decompose user needs, generate cascade labeling results, and then map the cascade labeling results to corresponding user needs, that is, to pre-set fields and behaviors, etc. Since manual labeling is not required, it saves manpower and time costs, improves processing efficiency, and improves the accuracy of processing results.

[0050] Among them, the cascade marking result can be shown as follows:

[0051] Problem Solving Calculations -> Chemistry Problem Solving -> Preparation of Potassium Permanganate;

[0052] Counting -> Entity Recognition -> Bird Counting;

[0053] Text recognition->image text to table;

[0054] Knowledge Questions and Answers -> Medical Knowledge -> Functions of Medical Devices;

[0055] Language translation->English to Chinese->Professional terminology.

[0056] Taking the cascade labeling result of "problem-solving calculation->chemical problem-solving->potassium permanganate preparation" as an example, its level is 3, including the three demand labels of "problem-solving calculation", "chemical problem-solving" and "potassium permanganate preparation".

[0057] In some embodiments of the present disclosure, a method for respectively generating cascade labeling results corresponding to each first training data may include: for each first training data, respectively performing the following processing: respectively restating the content of the conversation information and the answer in the first training data, determining the obtained conversation restatement results and the answer restatement results as input information, and generating the cascade labeling results corresponding to the first training data based on the input information and the cascade labeling model.

[0058] In addition, in some embodiments of the present disclosure, a method for generating a cascade marking result corresponding to the first training data based on the input information and the cascade marking model may include: inputting the input information and a pre-designed prompt word into the cascade marking model to obtain an output cascade marking result, wherein the prompt word includes: at least M representative cascade marking result examples, where M is a positive integer greater than 1.

[0059] Since subsequent domains and behaviors are all based on the output of cascade labeling results, we hope that the amount of information and the degree of abstraction contained in the cascade labeling results can be just right. To achieve this effect, we must not only prepare sufficiently accurate input information for the cascade labeling model, but also accurately express our expectations for its output results in the prompt words.

[0060] Accordingly, the method of restating the content and introducing the answer can be adopted, that is, for any first training data, the content of the dialogue information and the answer therein can be restated respectively, so as to obtain the dialogue restatement result and the answer restatement result, and then the dialogue restatement result and the answer restatement result can be determined as the input information. In addition, in terms of prompt word design, there is no need to agree on the total number of cascade labeling options, that is, there is no need to exhaustively list all cascade labeling situations, but only to provide sufficiently rich and representative cascade labeling result examples, and then these cascade labeling result examples can be used to guide the cascade labeling model to perform cascade labeling.

[0061] The specific value of M can be determined according to actual needs. The M cascade marking result examples can be manually set. In addition, for any two cascade marking result examples, the number of levels included therein can be the same or different.

[0062] In addition, there is no restriction on how to restate the content of the dialogue information and answers. For example, for a picture, the picture content description information can be extracted from it, and the picture content description information can be determined as the picture content restatement result. Among them, the picture content description information can be extracted using a large model, or the image caption generation technology can be used to extract the picture content description information.

[0063] By restating the content of the picture, the core information of the picture can be obtained. Especially when the picture contains a lot of content, directly inputting the picture may make it difficult for the model to grasp the key points. Accordingly, the core information can be extracted through content packaging processing, thereby improving the subsequent processing efficiency and the accuracy of the processing results. For example, for a picture of Suzhou gardens, the corresponding picture content description information can be: The picture shows a corner of Suzhou gardens, where stone bridges, rockery and pools complement each other, and pavilions and towers appear quiet and elegant in the sun, showing the unique charm of Jiangnan gardens.

[0064] For another example, for a question, the question can be restated to obtain a restatement result (a restatement result of the question content). The large model can be used to restate the question. For example, if the question is: Please help me write a poem to describe the beautiful scenery in the picture, the corresponding restatement result can be: The user hopes to obtain a poem describing the beautiful scenery to express the beauty and tranquility of nature or the environment.

[0065] In practical applications, the expression of the question may not be smooth or clear due to some reasons. The question can be restated to facilitate the model's understanding of the question. In addition, for some questions that are very long but have no corresponding pictures or the picture information is not helpful for answering the questions, overly lengthy questions may affect the model's understanding of the question. This situation can also be solved by restating the question.

[0066] For another example, for an answer, the answer can be restated to obtain the answer restatement result (result of restatement of the answer content). Among them, the large model can be used to restate the answer. For example, the answer is: The winding path leads to a secluded place, the bamboo shadows sway lightly, the pavilions and waterside pavilions are quiet, and the painted bridge reflects the clear waves. The corresponding answer restatement result can be: This poem depicts the beautiful scenery of Suzhou gardens. Through delicate descriptions, it shows the winding paths, bamboo shadows, pavilions, flowing water, rockery and other elements in the gardens, expressing the poetic atmosphere of beauty in all seasons and a fairyland on earth.

[0067] After respectively obtaining the cascade labeling results corresponding to each first training data, the user needs corresponding to each first training data can be determined according to the corresponding cascade labeling results, that is, the cascade labeling results are associated with pre-set fields and behaviors. Two association methods are provided in the scheme described in the present disclosure, which are introduced below respectively.

[0068] 1) Method 1

[0069] In some embodiments of the present disclosure, high-frequency demand labels can be firstly screened out from the cascade labeling results corresponding to each first training data, and the high-frequency demand labels include: demand labels whose appearance frequency in the cascade labeling results corresponding to each first training data is greater than a first threshold, and then a demand mapping table can be constructed according to the high-frequency demand labels, and then the user demand corresponding to each first training data can be determined according to the cascade labeling results corresponding to each first training data and the demand mapping table. The specific value of the first threshold can be determined according to actual needs.

[0070] Assuming that there are 100,000 first training data in total, and assuming that each first training data corresponds to 3 cascaded demand tags, then for the 300,000 demand tags, the demand tags whose occurrence frequency is greater than the first threshold can be screened out. Assuming that a total of 10,000 demand tags are screened out, these 10,000 demand tags can be determined as high-frequency demand tags, and a demand mapping table can be constructed based on the high-frequency demand tags. Then, according to the cascade labeling results corresponding to each first training data and the demand mapping table, the user needs corresponding to each first training data can be determined respectively.

[0071] In some embodiments of the present disclosure, the demand mapping table may include: a first mapping table and a second mapping table. Accordingly, the method of constructing a demand mapping table according to the high-frequency demand tags may include: respectively obtaining the first mapping results corresponding to each high-frequency demand tag, the first mapping results including: a field that matches the corresponding high-frequency demand tag selected from a pre-generated field list, and using each high-frequency demand tag and the corresponding first mapping result to form a first mapping table; respectively obtaining the second mapping results corresponding to each high-frequency demand tag, the second mapping results including: a behavior that matches the corresponding high-frequency demand tag selected from a pre-generated behavior list, and using each high-frequency demand tag and the corresponding second mapping result to form a second mapping table.

[0072] For example, assuming that there are 10,000 high-frequency demand tags in total, the first mapping table may include these 10,000 high-frequency demand tags and the first mapping results corresponding to these 10,000 high-frequency demand tags respectively. For any high-frequency demand tag, its corresponding first mapping result may include a field (which may be one or more) manually selected from the field list that matches the high-frequency demand tag. If there is no matching field, the first mapping result corresponding to the high-frequency demand tag may be empty. The second mapping table may include these 10,000 high-frequency demand tags and the second mapping results corresponding to these 10,000 high-frequency demand tags respectively. For any high-frequency demand tag, its corresponding second mapping result may include a behavior (which may be one or more) manually selected from the behavior list that matches the high-frequency demand tag. If there is no matching behavior, the second mapping result corresponding to the high-frequency demand tag may be empty.

[0073] Through the above processing, the accuracy of the generated first mapping table and the second mapping table can be ensured, thereby laying a good foundation for subsequent processing.

[0074] In some embodiments of the present disclosure, user needs may include: overall needs, which may include domains and behaviors. Accordingly, according to the cascade labeling results corresponding to each first training data and the demand mapping table, the method of respectively determining the user needs corresponding to each first training data may include: for each demand label in the cascade labeling results corresponding to any first training data, respectively performing the following processing: determining the high-frequency demand label that best matches the demand label among each high-frequency demand label by string matching, determining the first mapping result corresponding to the best matching high-frequency demand label as the domain mapping result corresponding to the demand label, and determining the second mapping result corresponding to the best matching high-frequency demand label as the behavior mapping result corresponding to the demand label; determining the domain in the domain mapping result corresponding to each demand label as the domain corresponding to the first training data, and determining the behavior in the behavior mapping result corresponding to each demand label as the behavior corresponding to the first training data.

[0075] For example, assuming that the cascade labeling result corresponding to a first training data includes three requirement labels, namely requirement label 1, requirement label 2 and requirement label 3. For requirement label 1, the high-frequency requirement label that best matches the requirement label among the high-frequency requirement labels can be first determined by string matching, assuming it is high-frequency requirement label x. Then, the first mapping result corresponding to the high-frequency requirement label x can be determined as the domain mapping result corresponding to requirement label 1, and the second mapping result corresponding to the high-frequency requirement label x can be determined as the behavior mapping result corresponding to requirement label 1. In the same way, the domain mapping result and behavior mapping result corresponding to requirement label 2 and the domain mapping result and behavior mapping result corresponding to requirement label 3 can be obtained respectively, and then the domain (after deduplication) in the domain mapping results corresponding to requirement labels 1, requirement labels 2 and requirement labels 3 can be determined as the domain corresponding to the first training data, and the behavior (after deduplication) in the behavior mapping results corresponding to requirement labels 1, requirement labels 2 and requirement labels 3 can be determined as the behavior corresponding to the first training data.

[0076] It can be seen that by adopting the above processing method, based on the constructed demand mapping table, the field and behavior corresponding to each first training data can be determined efficiently and accurately through simple string matching and other methods.

[0077] 2) Method 2

[0078] In some embodiments of the present disclosure, user needs may include: overall needs, which may include fields and behaviors. Accordingly, according to the corresponding cascade labeling results, the method of separately determining the user needs corresponding to each first training data may include: for each demand label in the cascade labeling result corresponding to any first training data, separately performing the following processing: in response to determining that there is a field matching the demand label in a pre-generated field list, determining the matching field as the field corresponding to the demand label; in response to determining that there is a behavior matching the demand label in a pre-generated behavior list, determining the matching behavior as the behavior corresponding to the demand label; determining the field corresponding to each demand label as the field corresponding to the first training data, and determining the behavior corresponding to each demand label as the behavior corresponding to the first training data.

[0079] For example, assuming that the cascade labeling result corresponding to a first training data includes three requirement labels, namely requirement label a, requirement label b and requirement label c. For requirement label a, if it is determined that there is a field matching requirement label a in the field list, then the matching field can be determined as the field corresponding to requirement label a. If it is determined that there is a behavior matching requirement label a in the behavior list, then the matching behavior can be determined as the behavior corresponding to requirement label a. In the same way, the field and behavior corresponding to requirement label b and the field and behavior corresponding to requirement label c can be obtained respectively, and then the field corresponding to requirement label a, requirement label b and requirement label c (after deduplication) can be determined as the field corresponding to the first training data, and the behavior corresponding to requirement label a, requirement label b and requirement label c (after deduplication) can be determined as the behavior corresponding to the first training data.

[0080] By adopting the above processing method, for any first training data, the fields and behaviors corresponding to each demand label corresponding to the first training data can be directly determined respectively, and then the fields and behaviors corresponding to the first training data can be determined by combining the fields and behaviors corresponding to each demand label, which is very simple and convenient.

[0081] In some embodiments of the present disclosure, for any requirement label, the feature similarity between the requirement label and each field in the field list can be determined respectively based on the feature representation of the requirement label and the feature representation of each field in the field list. In response to determining that there are fields whose feature similarity meets the requirements, the fields that meet the requirements can be determined as fields that match the requirement label. Similarly, the feature similarity between the requirement label and each behavior in the behavior list can be determined respectively based on the feature representation of the requirement label and the feature representation of each behavior in the behavior list. In response to determining that there are behaviors whose feature similarity meets the requirements, the behaviors that meet the requirements can be determined as behaviors that match the requirement label.

[0082] The feature representation may refer to text features extracted using a text feature extraction model. Feature extraction may be performed for each requirement label corresponding to each first training data, each field in the field list, and each behavior in the behavior list, thereby obtaining a corresponding feature representation.

[0083] In some embodiments of the present disclosure, in response to determining that there is a field whose feature similarity meets the requirements for any requirement tag, the following may be included: in response to determining that the feature similarity with the largest value is greater than a second threshold, the field corresponding to the feature similarity with the largest value is determined as the field that meets the requirements; and in response to determining that there is a behavior whose feature similarity meets the requirements, the following may be included: in response to determining that the feature similarity with the largest value is greater than a third threshold, the behavior corresponding to the feature similarity with the largest value is determined as the behavior that meets the requirements.

[0084] The specific values ​​of the second threshold and the third threshold can be determined according to actual needs, and the values ​​of the second threshold and the third threshold can be the same or different.

[0085] Taking the above-mentioned requirement label a as an example, the feature similarities between the requirement label a and each field in the field list can be determined respectively according to the feature representation of the requirement label a and the feature representation of each field in the field list, and then the feature similarity with the largest value can be selected from each feature similarity. If it is determined that the feature similarity with the largest value is greater than the second threshold, the field corresponding to the feature similarity with the largest value can be determined as the field that meets the requirements, that is, determined as the field corresponding to the requirement label a. In addition, according to the feature representation of the requirement label a and the feature representation of each behavior in the behavior list, the feature similarities between the requirement label a and each behavior in the behavior list can be determined respectively, and then the feature similarity with the largest value can be selected from each feature similarity. If it is determined that the feature similarity with the largest value is greater than the third threshold, the behavior corresponding to the feature similarity with the largest value can be determined as the behavior that meets the requirements, that is, determined as the behavior corresponding to the requirement label a.

[0086] In the above processing method, for any demand label, the corresponding field and behavior can be determined based on feature similarity, which is simple and convenient to implement and ensures the accuracy of the processing results.

[0087] After respectively acquiring the user requirements corresponding to the first training data, the second training data may be constructed according to the first training data and the corresponding user requirements.

[0088] In some embodiments of the present disclosure, the dialogue information may include: a question, or a question and a picture corresponding to the question. Accordingly, for any first training data, the dialogue information in the first training data, the answer in the first training data, and the user demand corresponding to the first training data may be respectively determined as the first candidate information, the second candidate information, and the third candidate information. Then, the second training data may be determined in the following manner: the second training data is composed of the first predetermined information and the third candidate information, the first predetermined information is the first candidate information or the first candidate information after optimization processing, and the optimization processing includes: restating the content of the question and / or picture in the first candidate information, or, the second training data is composed of the first predetermined information, the second predetermined information, and the third candidate information, the second predetermined information is the second candidate information or the answer restatement result obtained after restating the content of the second candidate information.

[0089] That is, the first candidate information and the third candidate information can be used to form the second training data, or the first candidate information, the second candidate information and the third candidate information can be used to form the second training data, or the first candidate information is optimized and the optimized first candidate information and the third candidate information can be used to form the second training data, or the optimized first candidate information, the second candidate information and the third candidate information can be used to form the second training data, or the second candidate information is restated to obtain a restated answer result, and the first candidate information, the restated answer result and the third candidate information can be used to form the second training data, or the optimized first candidate information, the restated answer result and the third candidate information can be used to form the second training data.

[0090] That is to say, in addition to user needs, the second training data may only include dialogue information. However, considering that the effect of such use may not necessarily meet expectations, the intelligent labeling model can be helped to understand user needs by introducing answers. In addition, the content of one or more of the pictures, questions and answers can be restated. Through content restatement, the user's needs can be expressed more simply. Moreover, for some overly complex answers and pictures, the length of the restatement results can be constrained during the restatement so that the intelligent labeling model can pay more attention to core information.

[0091] The constructed second training data can be used to form a second data set. In some embodiments of the present disclosure, the user requirements may also include: segmented scenario requirements, where the segmented scenario requirements include detailed requirement information after refining the overall requirements. Accordingly, second training data that meets the requirements can also be selected from the second data set as target training data, and the segmented scenario requirements can be added to the user requirements of the target training data.

[0092] Specifically, in some embodiments of the present disclosure, second training data that meets the requirements is selected from the second data set as the target training data, and a method of adding segmented scenario requirements to the user requirements of the target training data may include: in response to determining that the field corresponding to any second training data includes a field that meets the following requirements: the field is located in a pre-generated first segmented scenario mapping table, and the first segmented scenario mapping table includes: some fields selected from the field list and the segmented requirements corresponding to each field, the field that meets the requirements is determined as the target field, and the second training data is determined as the target training data, and the segmented scenario requirements corresponding to the target field are obtained, and the segmented scenario requirements are based on the target The segmented scenario requirements are generated according to the segmented requirements corresponding to the domain, the segmented scenario requirements are added to the user requirements of the target training data, and, in response to determining that the behavior corresponding to any second training data includes a behavior that meets the following requirements: the behavior is located in a pre-generated second segmented scenario mapping table, the second segmented scenario mapping table includes: some behaviors screened out from the behavior list and the segmented requirements corresponding to each behavior, the behavior that meets the requirements is determined as the target behavior, and the second training data is determined as the target training data, the segmented scenario requirements corresponding to the target behavior are obtained, the segmented scenario requirements are generated according to the segmented requirements corresponding to the target behavior, and the segmented scenario requirements are added to the user requirements of the target training data.

[0093] For example, a second training data corresponds to two behaviors, namely behavior 1 and behavior 2, and behavior 1 is located in the second segmented scenario mapping table. Then, behavior 1 can be determined as the target behavior, and the second training data can be determined as the target training data. After that, the segmented scenario requirements corresponding to the target behavior can be obtained. For example, if the target behavior is "translation", the corresponding segmented scenario requirements are "source language" and "target language", and then the segmented scenario requirements can be added to the user requirements of the target training data. In this way, the user requirements of the target training data will include both overall requirements (domains and behaviors) and segmented scenario requirements.

[0094] There is no restriction on how to determine the segmented scenario requirements corresponding to the target behavior. For example, it can be generated by manual annotation, or the developer can design the generation method according to actual needs.

[0095] Through the above processing, the data content in the second data set can be further improved. Afterwards, the second training data in the second data set can be used to train the intelligent labeling model, thereby improving the model training effect, so that the intelligent labeling model can learn more demand judgment capabilities, and thus meet the usage requirements of different scenarios.

[0096] Figure 2 Flow chart of an embodiment of the intelligent marking method described in the present disclosure. Figure 2 As shown, the following specific implementation methods are included.

[0097] In step 201, third training data is obtained, where the third training data includes dialogue information and answers corresponding to the dialogue information.

[0098] In step 202, the user needs corresponding to the third training data are generated using the intelligent labeling model. The intelligent labeling model is trained using the second training data in the second data set. The second training data are constructed based on the first training data in the first data set and the user needs corresponding to the first training data. The first training data includes dialogue information and answers corresponding to the dialogue information.

[0099] The third training data may be training data in a third data set used to train a certain large model. Accordingly, by adopting the scheme described in the above method embodiment, the intelligent labeling model may be used to label the third training data, that is, to generate user requirements corresponding to the third training data, so that the developer of the large model can have a better understanding of the distribution of user requirements in the third data set, so as to facilitate the developer to adjust and optimize the third data set. Figure 1 The intelligent labeling model trained by the method shown.

[0100] In some embodiments of the present disclosure, the dialogue information may include: questions, or questions and pictures corresponding to the questions. The method of using the intelligent labeling model to generate user needs corresponding to the third training data may include: determining the dialogue information and answers in the third training data as the first target information and the second target information respectively, inputting the third predetermined information into the intelligent labeling model to obtain the output user needs, the third predetermined information is the first target information or the first target information after optimization processing, and the optimization processing includes: restating the content of the questions and / or pictures in the first target information, or, inputting the third predetermined information and the fourth predetermined information into the intelligent labeling model to obtain the output user needs, the fourth predetermined information is the second target information or the answer restatement result obtained after restating the content of the second target information.

[0101] That is, the first target information can be input into the intelligent labeling model to obtain the output user demand, or the first target information and the second target information can be input into the intelligent labeling model to obtain the output user demand, or the first target information can be optimized and the optimized first target information can be input into the intelligent labeling model to obtain the output user demand, or the optimized first target information and the second target information can be input into the intelligent labeling model to obtain the output user demand, or the second target information can be restated to obtain the restated answer result, and the first target information and the restated answer result can be input into the intelligent labeling model to obtain the output user demand, or the optimized first target information and the restated answer result can be input into the intelligent labeling model to obtain the output user demand.

[0102] The specific information input into the smart labeling model may be determined according to the second training data used in training the smart labeling model, that is, the input during training and use remains consistent.

[0103] In addition, in some embodiments of the present disclosure, user needs may include: overall needs, the overall needs include fields and behaviors, or user needs may include: overall needs and segmented scenario needs, the segmented scenario needs include refined demand information after refining the overall needs.

[0104] That is to say, for the third training data, the intelligent labeling model can not only output the overall requirements, but also the segmented scenario requirements. Whether the segmented scenario requirements need to be output is determined by the intelligent labeling model based on the learned capabilities.

[0105] Through the above processing, it is possible to simultaneously realize the domain behavior demand judgment of open scenarios and the customized demand judgment of segmented scenarios in a unified intelligent labeling model, thereby meeting the usage requirements of different scenarios.

[0106] Combined with the above introduction, Figure 3 The overall implementation process of the intelligent marking method disclosed in this disclosure shows the requirements. Figure 3As shown, it is assumed that the third training data includes pictures, questions and answers, and the large model is used to restate the contents of the pictures, questions and answers, respectively, so as to obtain a first content restatement result (the content restatement result of the picture), a second content restatement result (the content restatement result of the question) and a third content restatement result (the content restatement result of the answer), respectively. Then, the first content restatement result, the second content restatement result, the third content restatement result and a predetermined prompt word (not shown in the figure to simplify the drawing) can be input into the intelligent labeling model to obtain the user needs corresponding to the third training data output by the intelligent labeling model. For example, the output user needs may include overall needs and segmented scene needs, wherein the field in the overall need may be "poetry, the behavior may be "creation", and the segmented scene needs may include "genre: **", "scene: **", "style: **" and "length: **", etc.

[0107] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0108] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0109] Figure 4 The components and structures of the intelligent marking model training device embodiment 400 described in the present disclosure are shown in FIG. Figure 4 As shown, it includes: a first processing module 401, a second processing module 402 and a model training module 403.

[0110] The first processing module 401 is used to obtain a first data set consisting of first training data, where the first training data includes dialogue information and answers corresponding to the dialogue information.

[0111] The second processing module 402 is used to generate user requirements corresponding to each first training data, and construct second training data according to the first training data and the corresponding user requirements.

[0112] The model training module 403 is used to use the constructed second training data to form a second data set, and use the second data set to train the intelligent labeling model, the intelligent labeling model is used to generate user needs corresponding to the third training data, and the third training data includes dialogue information and answers corresponding to the dialogue information.

[0113] By adopting the scheme described in the above-mentioned device embodiment, a second data set can be constructed based on a first data set composed of first training data, and then an intelligent labeling model can be trained based on the second training data in the second data set. The intelligent labeling model can then be used to perform demand labeling on the third training data, that is, to generate user needs corresponding to the third training data. The third training data can be the training data in the third data set used to train a certain large model. Accordingly, the developers of the large model can have a better understanding of the distribution of user needs in the third data set, so as to facilitate the developers to adjust and optimize the third data set.

[0114] For each first training data in the first data set acquired by the first processing module 401 , the second processing module 402 may generate corresponding user requirements respectively.

[0115] In some embodiments of the present disclosure, the second processing module 402 can respectively generate cascade labeling results corresponding to each first training data, each cascade labeling result includes at least two cascaded demand labels, and then the user needs corresponding to each first training data can be determined according to the corresponding cascade labeling results.

[0116] In some embodiments of the present disclosure, the method in which the second processing module 402 generates cascade labeling results corresponding to each first training data may include: performing the following processing on each first training data: restating the content of the conversation information and the answer in the first training data, determining the obtained conversation restatement results and the answer restatement results as input information, and generating the cascade labeling results corresponding to the first training data based on the input information and the cascade labeling model.

[0117] In addition, in some embodiments of the present disclosure, the way in which the second processing module 402 generates the cascade marking result corresponding to the first training data based on the input information and the cascade marking model may include: inputting the input information and the pre-designed prompt words into the cascade marking model to obtain the output cascade marking result, wherein the prompt words include: at least M representative cascade marking result examples, where M is a positive integer greater than 1.

[0118] After respectively obtaining the cascade labeling results corresponding to each first training data, the second processing module 402 can determine the user needs corresponding to each first training data according to the corresponding cascade labeling results, that is, associate the cascade labeling results with pre-set fields and behaviors. Two association methods are provided in the solution described in the present disclosure, which are introduced below.

[0119] In some embodiments of the present disclosure, the second processing module 402 can filter out high-frequency demand labels from the cascade labeling results corresponding to each first training data, and the high-frequency demand labels include: demand labels whose appearance frequency in the cascade labeling results corresponding to each first training data is greater than a first threshold, and a demand mapping table is constructed based on the high-frequency demand labels. Based on the cascade labeling results corresponding to each first training data and the demand mapping table, the user demands corresponding to each first training data are respectively determined.

[0120] In some embodiments of the present disclosure, the demand mapping table may include: a first mapping table and a second mapping table. Accordingly, the second processing module 402 may construct a demand mapping table according to the high-frequency demand tags, which may include: respectively obtaining the first mapping results corresponding to each high-frequency demand tag, the first mapping results including: a field that matches the corresponding high-frequency demand tag selected from a pre-generated field list, and using each high-frequency demand tag and the corresponding first mapping result to form a first mapping table; respectively obtaining the second mapping results corresponding to each high-frequency demand tag, the second mapping results including: a behavior that matches the corresponding high-frequency demand tag selected from a pre-generated behavior list, and using each high-frequency demand tag and the corresponding second mapping result to form a second mapping table.

[0121] In some embodiments of the present disclosure, user needs may include: overall needs, which may include domains and behaviors. Accordingly, the second processing module 402 may determine the user needs corresponding to each first training data according to the cascade labeling results corresponding to each first training data and the demand mapping table, which may include: for each demand label in the cascade labeling results corresponding to any first training data, the following processing is performed respectively: the high-frequency demand label that best matches the demand label among each high-frequency demand label is determined by string matching, and the first mapping result corresponding to the best matching high-frequency demand label is determined as the domain mapping result corresponding to the demand label, and the second mapping result corresponding to the best matching high-frequency demand label is determined as the behavior mapping result corresponding to the demand label; the domain in the domain mapping result corresponding to each demand label is determined as the domain corresponding to the first training data, and the behavior in the behavior mapping result corresponding to each demand label is determined as the behavior corresponding to the first training data.

[0122] In addition, in some embodiments of the present disclosure, user needs may include: overall needs, and overall needs may include fields and behaviors. Accordingly, the second processing module 402 may determine the user needs corresponding to each first training data according to the corresponding cascade labeling results, which may include: for each demand label in the cascade labeling results corresponding to any first training data, the following processing is performed respectively: in response to determining that there is a field matching the demand label in the pre-generated field list, determining the matching field as the field corresponding to the demand label; in response to determining that there is a behavior matching the demand label in the pre-generated behavior list, determining the matching behavior as the behavior corresponding to the demand label; determining the field corresponding to each demand label as the field corresponding to the first training data, and determining the behavior corresponding to each demand label as the behavior corresponding to the first training data.

[0123] In some embodiments of the present disclosure, for any requirement label, the second processing module 402 can determine the feature similarity between the requirement label and each field in the field list based on the feature representation of the requirement label and the feature representation of each field in the field list. In response to determining that there are fields whose feature similarity meets the requirements, the fields that meet the requirements can be determined as fields that match the requirement label. Similarly, the feature similarity between the requirement label and each behavior in the behavior list can be determined based on the feature representation of the requirement label and the feature representation of each behavior in the behavior list. In response to determining that there are behaviors whose feature similarity meets the requirements, the behaviors that meet the requirements can be determined as behaviors that match the requirement label.

[0124] In some embodiments of the present disclosure, for any requirement tag, the second processing module 402 may determine, with respect to the feature similarities between the requirement tag and each field in the field list, in response to determining that the feature similarity with the largest value is greater than the second threshold, the field corresponding to the feature similarity with the largest value may be determined as the field that meets the requirements; and with respect to the feature similarities between the requirement tag and each behavior in the behavior list, in response to determining that the feature similarity with the largest value is greater than the third threshold, the behavior corresponding to the feature similarity with the largest value may be determined as the behavior that meets the requirements.

[0125] After respectively acquiring the user requirements corresponding to each first training data, the second processing module 402 may further construct each second training data according to each first training data and the corresponding user requirements.

[0126] In some embodiments of the present disclosure, the dialogue information may include: questions, or questions and pictures corresponding to the questions. Accordingly, for any first training data, the second processing module 402 may determine the dialogue information in the first training data, the answers in the first training data, and the user needs corresponding to the first training data as first candidate information, second candidate information, and third candidate information, respectively. Then, the second training data may be determined in the following manner: the second training data is composed of the first predetermined information and the third candidate information, the first predetermined information is the first candidate information or the first candidate information after optimization processing, and the optimization processing includes: restating the content of the question and / or picture in the first candidate information, or, the second training data is composed of the first predetermined information, the second predetermined information, and the third candidate information, the second predetermined information is the second candidate information or the answer restatement result obtained after restating the content of the second candidate information.

[0127] That is, the first candidate information and the third candidate information can be used to form the second training data, or the first candidate information, the second candidate information and the third candidate information can be used to form the second training data, or the first candidate information is optimized and the optimized first candidate information and the third candidate information can be used to form the second training data, or the optimized first candidate information, the second candidate information and the third candidate information can be used to form the second training data, or the second candidate information is restated to obtain a restated answer result, and the first candidate information, the restated answer result and the third candidate information can be used to form the second training data, or the optimized first candidate information, the restated answer result and the third candidate information can be used to form the second training data.

[0128] The constructed second training data can be used to form a second data set. In some embodiments of the present disclosure, user requirements may also include: segmented scenario requirements, where the segmented scenario requirements include refined requirement information after refining the overall requirements. Accordingly, the model training module 403 may also select second training data that meets the requirements from the second data set as target training data, and may add segmented scenario requirements to the user requirements of the target training data.

[0129] Specifically, in some embodiments of the present disclosure, the model training module 403 selects second training data that meets the requirements from the second data set as target training data, and adds segmented scenario requirements to the user requirements of the target training data. The method may include: in response to determining that the field corresponding to any second training data includes a field that meets the following requirements: the field is located in a pre-generated first segmented scenario mapping table, and the first segmented scenario mapping table includes: some fields selected from the field list and the segmented requirements corresponding to each field, the field that meets the requirements is determined as the target field, and the second training data is determined as the target training data, and the segmented scenario requirements corresponding to the target field are obtained, and the segmented scenario requirements are Generated according to the segmentation requirements corresponding to the target field, the segmentation scenario requirements are added to the user requirements of the target training data, and, in response to determining that the behavior corresponding to any second training data includes a behavior that meets the following requirements: the behavior is located in a pre-generated second segmentation scenario mapping table, the second segmentation scenario mapping table includes: some behaviors screened out from the behavior list and the segmentation requirements corresponding to each behavior, the behavior that meets the requirements is determined as the target behavior, and the second training data is determined as the target training data, the segmentation scenario requirements corresponding to the target behavior are obtained, the segmentation scenario requirements are generated according to the segmentation requirements corresponding to the target behavior, and the segmentation scenario requirements are added to the user requirements of the target training data.

[0130] Figure 5 The components and structures of the marking device embodiment 500 described in the present disclosure are shown in FIG. Figure 5 As shown, it includes: a data acquisition module 501 and a marking processing module 502.

[0131] The data acquisition module 501 is used to acquire third training data, where the third training data includes dialogue information and answers corresponding to the dialogue information.

[0132] The labeling processing module 502 is used to generate user requirements corresponding to third training data using an intelligent labeling model. The intelligent labeling model is obtained by training using each second training data in the second data set. Each second training data is constructed based on each first training data in the first data set and the user requirements corresponding to each first training data. The first training data includes dialogue information and answers corresponding to the dialogue information.

[0133] The third training data may be training data in a third data set used to train a certain large model. Accordingly, by adopting the scheme described in the above-mentioned device embodiment, an intelligent labeling model may be used to perform demand labeling on the third training data, that is, to generate user demands corresponding to the third training data, thereby enabling the developers of the large model to have a better understanding of the distribution of user demands in the third data set, so as to facilitate the developers to adjust and optimize the third data set.

[0134] In some embodiments of the present disclosure, the dialogue information may include: questions, or questions and pictures corresponding to the questions. The way in which the labeling processing module 502 uses the intelligent labeling model to generate user needs corresponding to the third training data may include: determining the dialogue information and answers in the third training data as the first target information and the second target information respectively, inputting the third predetermined information into the intelligent labeling model to obtain output user needs, the third predetermined information is the first target information or the first target information after optimization processing, and the optimization processing includes: restating the content of the questions and / or pictures in the first target information, or, inputting the third predetermined information and the fourth predetermined information into the intelligent labeling model to obtain output user needs, the fourth predetermined information is the second target information or the answer restatement result obtained after restating the content of the second target information.

[0135] That is, the first target information can be input into the intelligent labeling model to obtain the output user demand, or the first target information and the second target information can be input into the intelligent labeling model to obtain the output user demand, or the first target information can be optimized and the optimized first target information can be input into the intelligent labeling model to obtain the output user demand, or the optimized first target information and the second target information can be input into the intelligent labeling model to obtain the output user demand, or the second target information can be restated to obtain the restated answer result, and the first target information and the restated answer result can be input into the intelligent labeling model to obtain the output user demand, or the optimized first target information and the restated answer result can be input into the intelligent labeling model to obtain the output user demand.

[0136] In addition, in some embodiments of the present disclosure, user needs may include: overall needs, the overall needs include fields and behaviors, or user needs may include: overall needs and segmented scenario needs, the segmented scenario needs include refined demand information after refining the overall needs.

[0137] That is to say, for the third training data, the labeling processing module 502 can output not only the overall requirements but also the segmented scenario requirements. Whether the segmented scenario requirements need to be output is determined by the intelligent labeling model based on the learned capabilities.

[0138] Figure 4 and Figure 5 The specific working process of the device embodiment shown can refer to the relevant description in the aforementioned method embodiment, which will not be repeated here.

[0139] The scheme disclosed in the present invention can be applied to the field of artificial intelligence, especially computer vision, deep learning, and large models. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0140] The conversation information and answers in the embodiments described in the present disclosure are not targeted at a specific user and do not reflect the personal information of a specific user. In addition, the executor of the method described in the present disclosure can obtain the conversation information in various public, legal and compliant ways, such as obtaining it from the user with the user's authorization.

[0141] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0142] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0143] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0144] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0145] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0146] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0147] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0149] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0151] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0152] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0154] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A smart labeling model training method, comprising: Acquire a first data set consisting of first training data, where the first training data includes dialogue information and answers corresponding to the dialogue information; Generating user requirements corresponding to each first training data respectively, and constructing second training data according to the first training data and the corresponding user requirements; The constructed second training data is used to form a second data set, and the second data set is used to train an intelligent labeling model, wherein the intelligent labeling model is used to generate user requirements corresponding to third training data, wherein the third training data includes dialogue information and answers corresponding to the dialogue information.

2. The method according to claim 1, wherein: The generating of user requirements corresponding to each first training data includes: Generate cascade labeling results corresponding to the first training data respectively, each cascade labeling result includes at least two cascade requirement labels; According to the corresponding cascade labeling results, the user needs corresponding to each first training data are determined respectively.

3. The method according to claim 2, wherein: The generating of the cascade labeling results corresponding to the first training data respectively includes: For each first training data, the conversation information and the answer in the first training data are restated respectively, and the obtained conversation restatement results and answer restatement results are determined as input information, and the cascade labeling results corresponding to the first training data are generated according to the input information and the cascade labeling model.

4. The method according to claim 3, wherein: Generating the cascade marking result corresponding to the first training data according to the input information and the cascade marking model includes: The input information and the pre-designed prompt words are input into the cascade marking model to obtain the output cascade marking result, wherein the prompt words include: at least M representative cascade marking result examples, where M is a positive integer greater than 1.

5. The method according to claim 2, wherein: Determining the user requirements corresponding to each first training data according to the corresponding cascade labeling results includes: Filter out high-frequency demand tags from the cascade labeling results corresponding to each first training data, wherein the high-frequency demand tags include: demand tags whose appearance frequency in the cascade labeling results corresponding to each first training data is greater than a first threshold; A demand mapping table is constructed according to the high-frequency demand labels, and user demands corresponding to each first training data are determined according to the cascade labeling results corresponding to each first training data and the demand mapping table.

6. The method according to claim 5, wherein: The demand mapping table includes: a first mapping table and a second mapping table; The constructing a demand mapping table according to the high-frequency demand label includes: Obtaining first mapping results corresponding to each high-frequency demand tag respectively, wherein the first mapping results include: fields that match the corresponding high-frequency demand tag selected from a pre-generated field list; and forming the first mapping table using each high-frequency demand tag and the corresponding first mapping result; The second mapping results corresponding to each high-frequency demand label are obtained respectively, wherein the second mapping results include: behaviors matching the corresponding high-frequency demand labels selected from a pre-generated behavior list; and the second mapping table is composed of each high-frequency demand label and the corresponding second mapping results.

7. The method according to claim 6, wherein: The user needs include: overall needs, which include domains and behaviors; The determining, according to the cascade marking results corresponding to the first training data and the demand mapping table, the user demand corresponding to each first training data comprises: For each demand label in the cascade labeling result corresponding to any first training data, the high-frequency demand label that best matches the demand label is determined by string matching, and the first mapping result corresponding to the best-matching high-frequency demand label is determined as the domain mapping result corresponding to the demand label, and the second mapping result corresponding to the best-matching high-frequency demand label is determined as the behavior mapping result corresponding to the demand label; the domain in the domain mapping result corresponding to each demand label is determined as the domain corresponding to the first training data, and the behavior in the behavior mapping result corresponding to each demand label is determined as the behavior corresponding to the first training data.

8. The method according to claim 2, wherein: The user needs include: overall needs, which include domains and behaviors; Determining the user requirements corresponding to each first training data according to the corresponding cascade labeling results includes: For each requirement label in the cascade labeling result corresponding to any first training data, in response to determining that there is a field matching the requirement label in a pre-generated field list, the matched field is determined as the field corresponding to the requirement label; in response to determining that there is a behavior matching the requirement label in a pre-generated behavior list, the matched behavior is determined as the behavior corresponding to the requirement label; the field corresponding to each requirement label is determined as the field corresponding to the first training data, and the behavior corresponding to each requirement label is determined as the behavior corresponding to the first training data.

9. The method according to claim 8, wherein: The response to determining that there is a domain matching the requirement tag in the pre-generated domain list includes: Determining the feature similarity between the requirement label and each field in the field list respectively according to the feature representation of the requirement label and the feature representation of each field in the field list, and in response to determining that there is a field whose feature similarity meets the requirement, determining the field that meets the requirement as a field that matches the requirement label; The response to determining that there is a behavior matching the requirement tag in the pre-generated behavior list includes: Based on the feature representation of the requirement label and the feature representation of each behavior in the behavior list, the feature similarities between the requirement label and each behavior in the behavior list are determined respectively. In response to determining that there is a behavior whose feature similarity meets the requirements, the behavior that meets the requirements is determined as a behavior that matches the requirement label.

10. The method according to claim 9, wherein: The responding to determining that there is a field whose feature similarity meets the requirement includes: responding to determining that the feature similarity with the largest value is greater than a second threshold, determining the field corresponding to the feature similarity with the largest value as the field that meets the requirement; The responding to determining that there is a behavior whose feature similarity meets the requirement includes: in response to determining that the feature similarity with the largest value is greater than a third threshold, determining the behavior corresponding to the feature similarity with the largest value as the behavior that meets the requirement.

11. The method according to claim 7 or 8, wherein: The dialogue information includes: a question, or a question and a picture corresponding to the question; The constructing the second training data according to the first training data and the corresponding user needs includes: respectively determining the dialogue information in the first training data, the answers in the first training data, and the user requirements corresponding to the first training data as first to-be-selected information, second to-be-selected information, and third to-be-selected information; The second training data is composed of the first predetermined information and the third candidate information, wherein the first predetermined information is the first candidate information or the first candidate information after optimization processing, and the optimization processing includes: restating the content of the question and / or the picture in the first candidate information, or, the second training data is composed of the first predetermined information, the second predetermined information and the third candidate information, wherein the second predetermined information is the second candidate information or the answer restatement result obtained after restating the content of the second candidate information.

12. The method according to claim 11, wherein: The user demand also includes: segmented scenario demand, wherein the segmented scenario demand includes detailed demand information after refining the overall demand; The method also includes: before using the second data set to train the intelligent labeling model, selecting second training data that meets the requirements from the second data set as target training data, and adding the segmented scenario requirements to the user requirements of the target training data.

13. The method according to claim 12, wherein: The selecting the second training data meeting the requirements from the second data set as the target training data, and adding the segmented scenario requirements to the user requirements of the target training data includes: In response to determining that the field corresponding to any second training data includes a field that meets the following requirements: the field is located in a pre-generated first segmented scene mapping table, the first segmented scene mapping table includes: some fields screened from the field list and segmented requirements corresponding to each field, the field that meets the requirements is determined as a target field, and the second training data is determined as the target training data, a segmented scene requirement corresponding to the target field is obtained, the segmented scene requirement is generated according to the segmented requirement corresponding to the target field, and the segmented scene requirement is added to the user requirement of the target training data; And, in response to determining that the behavior corresponding to any second training data includes a behavior that meets the following requirements: the behavior is located in a pre-generated second segmented scenario mapping table, and the second segmented scenario mapping table includes: some behaviors screened out from the behavior list and the segmented requirements corresponding to each behavior, the behavior that meets the requirements is determined as the target behavior, and the second training data is determined as the target training data, and the segmented scenario requirements corresponding to the target behavior are obtained, and the segmented scenario requirements are generated according to the segmented requirements corresponding to the target behavior, and the segmented scenario requirements are added to the user requirements of the target training data.

14. An intelligent marking method, comprising: Acquire third training data, wherein the third training data includes dialogue information and answers corresponding to the dialogue information; The user needs corresponding to the third training data are generated using an intelligent labeling model, wherein the intelligent labeling model is trained using the second training data in the second data set, and the second training data are constructed based on the first training data in the first data set and the user needs corresponding to the first training data, respectively, and the first training data include dialogue information and answers corresponding to the dialogue information.

15. The method according to claim 14, wherein: The dialogue information includes: a question, or a question and a picture corresponding to the question; The user demand corresponding to the third training data generated by using the intelligent labeling model includes: Determine the dialogue information and the answer in the third training data as the first target information and the second target information respectively; The third predetermined information is input into the intelligent labeling model to obtain the output user demand, the third predetermined information is the first target information or the first target information after optimization processing, the optimization processing includes: restating the content of the question and / or the picture in the first target information, or, the third predetermined information and the fourth predetermined information are input into the intelligent labeling model to obtain the output user demand, the fourth predetermined information is the second target information or the answer restatement result obtained after restating the content of the second target information.

16. The method according to claim 14, wherein: The user needs include: overall needs, which include domains and behaviors; Alternatively, the user demand includes: the overall demand and segmented scenario demand, and the segmented scenario demand includes refined demand information that is obtained by refining the overall demand.

17. An intelligent marking model training device, comprising: A first processing module, a second processing module and a model training module; The first processing module is used to obtain a first data set consisting of first training data, where the first training data includes dialogue information and answers corresponding to the dialogue information; The second processing module is used to generate user requirements corresponding to each first training data, and construct second training data according to the first training data and the corresponding user requirements; The model training module is used to use the constructed second training data to form a second data set, and use the second data set to train the intelligent labeling model, the intelligent labeling model is used to generate user needs corresponding to the third training data, and the third training data includes dialogue information and answers corresponding to the dialogue information.

18. The device according to claim 17, wherein: The second processing module generates cascade labeling results corresponding to each first training data respectively, each cascade labeling result includes at least two cascaded demand labels respectively, and determines the user demand corresponding to each first training data according to the corresponding cascade labeling results.

19. The device according to claim 18, wherein: The second processing module restates the conversation information and the answer in each first training data, respectively, determines the obtained conversation restatement results and answer restatement results as input information, and generates a cascade labeling result corresponding to the first training data based on the input information and the cascade labeling model.

20. The device according to claim 19, wherein The second processing module inputs the input information and the pre-designed prompt words into the cascade marking model to obtain the output cascade marking result, wherein the prompt words include: at least M representative cascade marking result examples, where M is a positive integer greater than 1.

21. The device according to claim 18, wherein The second processing module filters out high-frequency demand tags from the cascade labeling results corresponding to each first training data, and the high-frequency demand tags include: demand tags whose appearance frequency in the cascade labeling results corresponding to each first training data is greater than a first threshold, and constructs a demand mapping table based on the high-frequency demand tags, and determines the user demand corresponding to each first training data according to the cascade labeling results corresponding to each first training data and the demand mapping table.

22. The device according to claim 21, wherein The demand mapping table includes: a first mapping table and a second mapping table; The second processing module obtains the first mapping results corresponding to each high-frequency demand tag respectively, and the first mapping results include: the fields that match the corresponding high-frequency demand tags screened out from the pre-generated field list, and the first mapping table is composed of each high-frequency demand tag and the corresponding first mapping result, and the second mapping results corresponding to each high-frequency demand tag are obtained respectively, and the second mapping results include: the behaviors that match the corresponding high-frequency demand tags screened out from the pre-generated behavior list, and the second mapping table is composed of each high-frequency demand tag and the corresponding second mapping result.

23. The device according to claim 22, wherein: The user needs include: overall needs, which include domains and behaviors; The second processing module determines, for each requirement label in the cascade labeling result corresponding to any first training data, the high-frequency requirement label that best matches the requirement label among each high-frequency requirement label by string matching, determines the first mapping result corresponding to the best-matching high-frequency requirement label as the domain mapping result corresponding to the requirement label, and determines the second mapping result corresponding to the best-matching high-frequency requirement label as the behavior mapping result corresponding to the requirement label, determines the domain in the domain mapping results corresponding to each requirement label as the domain corresponding to the first training data, and determines the behavior in the behavior mapping results corresponding to each requirement label as the behavior corresponding to the first training data.

24. The device according to claim 18, wherein The user needs include: overall needs, which include domains and behaviors; The second processing module, for each requirement label in the cascade labeling result corresponding to any first training data, in response to determining that there is a field matching the requirement label in a pre-generated field list, determines the matched field as the field corresponding to the requirement label, in response to determining that there is a behavior matching the requirement label in a pre-generated behavior list, determines the matched behavior as the behavior corresponding to the requirement label, determines the field corresponding to each requirement label as the field corresponding to the first training data, and determines the behavior corresponding to each requirement label as the behavior corresponding to the first training data.

25. The device according to claim 24, wherein: The second processing module determines, for any requirement tag, the feature similarity between the requirement tag and each field in the field list based on the feature representation of the requirement tag and the feature representation of each field in the field list; in response to determining that there is a field whose feature similarity meets the requirement, the field that meets the requirement is determined as the field that matches the requirement tag; determines, based on the feature representation of the requirement tag and the feature representation of each behavior in the behavior list, the feature similarity between the requirement tag and each behavior in the behavior list; in response to determining that there is a behavior whose feature similarity meets the requirement, the behavior that meets the requirement is determined as the behavior that matches the requirement tag.

26. The device according to claim 25, wherein The second processing module determines, with respect to the feature similarities between the requirement label and each field in the field list, in response to determining that the feature similarity with the largest value is greater than a second threshold, the field corresponding to the feature similarity with the largest value is determined as the field that meets the requirements; and with respect to the feature similarities between the requirement label and each behavior in the behavior list, in response to determining that the feature similarity with the largest value is greater than a third threshold, the behavior corresponding to the feature similarity with the largest value is determined as the behavior that meets the requirements.

27. The device according to claim 23 or 24, wherein: The dialogue information includes: a question, or a question and a picture corresponding to the question; The second processing module determines the dialogue information in the first training data, the answer in the first training data, and the user demand corresponding to the first training data as the first candidate information, the second candidate information, and the third candidate information, respectively, and uses the first predetermined information and the third candidate information to form the second training data, the first predetermined information is the first candidate information or the first candidate information after optimization processing, the optimization processing includes: restating the content of the question and / or the picture in the first candidate information, or, using the first predetermined information, the second predetermined information and the third candidate information to form the second training data, the second predetermined information is the second candidate information or the answer restatement result obtained after restating the content of the second candidate information.

28. The device according to claim 27, wherein The user demand also includes: segmented scenario demand, wherein the segmented scenario demand includes detailed demand information after refining the overall demand; The model training module is further used to select second training data that meets the requirements from the second data set as target training data before using the second data set to train the intelligent labeling model, and add the segmented scenario requirements to the user requirements of the target training data.

29. The device according to claim 28, wherein The model training module responds to determining that the field corresponding to any second training data includes a field that meets the following requirements: the field is located in a pre-generated first segmented scenario mapping table, the first segmented scenario mapping table includes: some fields selected from the field list and the segmented requirements corresponding to each field, the field that meets the requirements is determined as the target field, and the second training data is determined as the target training data, the segmented scenario requirements corresponding to the target field are obtained, the segmented scenario requirements are generated according to the segmented requirements corresponding to the target field, and the segmented scenario requirements are added to the user requirements of the target training data, and, in response to determining that the behavior corresponding to any second training data includes a behavior that meets the following requirements: the behavior is located in a pre-generated second segmented scenario mapping table, the second segmented scenario mapping table includes: some behaviors selected from the behavior list and the segmented requirements corresponding to each behavior, the behavior that meets the requirements is determined as the target behavior, and the second training data is determined as the target training data, the segmented scenario requirements corresponding to the target behavior are obtained, the segmented scenario requirements are generated according to the segmented requirements corresponding to the target behavior, and the segmented scenario requirements are added to the user requirements of the target training data.

30. A marking device, comprising: Data acquisition module and marking processing module; The data acquisition module is used to acquire third training data, wherein the third training data includes dialogue information and answers corresponding to the dialogue information; The labeling processing module is used to generate user needs corresponding to the third training data using an intelligent labeling model, wherein the intelligent labeling model is trained using each second training data in the second data set, and each second training data is constructed based on each first training data in the first data set and the user needs corresponding to each first training data, and the first training data includes dialogue information and answers corresponding to the dialogue information.

31. The device according to claim 30, wherein The dialogue information includes: a question, or a question and a picture corresponding to the question; The labeling processing module determines the dialogue information and the answer in the third training data as the first target information and the second target information respectively, inputs the third predetermined information into the intelligent labeling model, and obtains the output user demand, the third predetermined information is the first target information or the first target information after optimization processing, and the optimization processing includes: restating the content of the question and / or the picture in the first target information, or, inputting the third predetermined information and the fourth predetermined information into the intelligent labeling model, and obtaining the output user demand, the fourth predetermined information is the second target information or the answer restatement result obtained after the content of the second target information is restated.

32. The device according to claim 30, wherein: The user needs include: overall needs, which include domains and behaviors; Alternatively, the user demand includes: the overall demand and segmented scenario demand, and the segmented scenario demand includes refined demand information that is obtained by refining the overall demand.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 16.

35. A computer program product, comprising a computer program / instructions, which implement the method according to any one of claims 1 to 16 when executed by a processor.

Images