Large language model training method, training data acquisition method, and intent recognition method

Abstract

The application discloses a large language model training method, a training data acquisition method and an intent recognition method, and relates to the technical field of artificial intelligence, and comprises the following steps: an electronic device acquires an original data set. The original data set comprises a plurality of dialogue data. The electronic device divides the dialogue data in the original data set into groups based on a preset data dimension, and obtains a plurality of groups. The electronic device samples data from the plurality of groups respectively, and takes the union of the sampled dialogue data as a training data set. The electronic device takes the training data set as input of a large language model to perform model training, and obtains a large language model after training. The original data set is grouped and sampled based on different data dimensions, and the data processing is performed, so that each training data set obtained is relatively balanced in each data dimension distribution, and the large language model obtained based on the training data set has a high accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for training a large language model, a method for acquiring training data, and a method for intent recognition. Background Technology

[0002] In dialogue scenarios based on large language models, user dialogue commands typically contain multiple intents and slots. The large language model is needed to accurately distribute the intents within the user's dialogue commands and extract the relevant slot content, thereby enabling appropriate response processing based on the intent and slot values. Therefore, training a highly accurate large language model is a crucial issue.

[0003] A common model training method is to randomly sample from the original dataset to obtain a training dataset. This training dataset is then input into an initial large language model for training. When the model reaches the preset performance indicators, the trained large language model is obtained.

[0004] However, in different dialogue scenarios, and even in the same dialogue scenario, the intentions and slots contained in different dialogue instructions are different in multiple dimensions. The above-mentioned random sampling of the original dataset to obtain the training dataset, and the model training based on this training dataset, results in a large language model with low accuracy. Summary of the Invention

[0005] This application provides a method for training a large language model, a method for acquiring training data, and a method for intent recognition. The method involves grouping and sampling the original dataset based on different data dimensions to obtain a training dataset with a relatively balanced distribution across all data dimensions. Training a large language model on this balanced training dataset results in a large language model with high accuracy in processing dialogue commands across all dimensions.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions.

[0007] Firstly, a language model training method is provided, in which a large language model is applied to human-computer dialogue scenarios. This method includes:

[0008] The electronic device acquires the raw dataset. This raw dataset includes multiple dialogue data sets, which are used to characterize the data contained in dialogues within a human-computer dialogue scenario.

[0009] The electronic device categorizes the dialogue data in the original dataset into multiple groups based on preset data dimensions. Each group corresponds to one data dimension or a combination of one or more data dimensions.

[0010] The electronic device samples data from multiple groups, and uses the combined sampled dialogue data as a training dataset. The electronic device then uses this training dataset as input to train a large language model, resulting in the trained large language model.

[0011] In this application, based on a pre-defined grouping and sampling strategy, the original dataset is grouped and sampled across multiple data dimensions to obtain a training dataset. The training dataset maintains a balanced distribution across multiple data dimensions. By training a large language model using this dataset, the resulting large language model exhibits excellent performance across multiple data dimensions, demonstrating high accuracy and recall.

[0012] In one possible implementation of the first aspect, the preset data dimensions include at least one of data type, intent classification, number of dialogue turns, and combination of non-empty slots. When the preset data dimensions include multiple data dimensions, each data dimension has a corresponding priority.

[0013] The electronic device categorizes the dialogue data in the original dataset into groups based on preset data dimensions, resulting in multiple groups, including:

[0014] The electronic device divides the dialogue data in the original dataset into groups according to the priority of each data dimension from highest to lowest, resulting in multiple groups.

[0015] In this application, different data dimensions can characterize the degree of refinement used to divide dialogue data, and setting the priority of different data dimensions can make grouping more effective.

[0016] In another possible implementation of the first aspect, the preset data dimensions include data type, intent category, number of dialogue turns, and non-empty slot combination, with data type having a higher priority than intent category, intent category having a higher priority than number of dialogue turns, and number of dialogue turns having a higher priority than non-empty slot combination.

[0017] The electronic device divides the dialogue data in the original dataset into groups according to the priority of each data dimension, from highest to lowest, resulting in multiple groups, including:

[0018] The electronic device categorizes the dialogue data in the original dataset into groups according to data type, resulting in the first group. Data types include at least two of the following: single turn, follow-up question, domain-specific, secondary execution, intent inheritance, and slot inheritance.

[0019] Under the same data type, the electronic device categorizes the dialogue data in each first group according to the intent classification to obtain the second group;

[0020] Under the same intent classification, the electronic device categorizes the dialogue data in each second group according to the number of dialogue turns, resulting in a third group;

[0021] With the same number of dialogue rounds, the electronic device divides the dialogue data in each third group into multiple groups according to the combination of non-empty slots.

[0022] In this application, the electronic device can group the dialogue data in the original dataset based on at least one preset data dimension, which can ensure that the constructed training dataset maintains a balanced distribution across multiple data dimensions. Training a large language model based on a balanced training dataset can improve the efficiency of model training.

[0023] In another possible implementation of the first aspect, the preset data dimension includes data types, which include a first data type and a second data type.

[0024] The electronic device samples data from multiple groups, and uses the combined set of sampled dialogue data as the training dataset, including:

[0025] The electronic device samples a first number of dialogue data from the group corresponding to the first data type, and samples a second number of dialogue data from the group corresponding to the second data type. The combined set of the sampled dialogue data is then used as the training dataset.

[0026] The first data type represents a larger quantity than the second data type, and the level of detail in the dialogue data represented by the first data type is lower than that represented by the second data type. The first data type includes single-turn dialogues, follow-up questions, or domain-specific dialogues, while the second data type includes secondary execution, intent inheritance, or slot inheritance.

[0027] In this application, the electronic device can group the dialogue data in the original dataset based on at least one preset data dimension, and sample a corresponding number of data for different data types. This improves the effectiveness of the sampled data while ensuring a balanced distribution of the training dataset across all data dimensions. Training a large language model based on a balanced training dataset can improve the efficiency of model training.

[0028] In another possible implementation of the first aspect, the method further includes:

[0029] The electronic device performs data correction processing on the training dataset to obtain the corrected training dataset.

[0030] The electronic device categorizes the dialogue data in the original dataset into groups based on preset data dimensions, resulting in multiple groups, including:

[0031] The electronic device divides the dialogue data in the training dataset after data correction and processing into multiple groups based on a preset data dimension.

[0032] In this application, data correction processing is performed on the original dataset to ensure the validity of the dialogue data in the original dataset (which can also be the training dataset). Model training based on the training dataset can improve the accuracy and recall of the large language model.

[0033] In another possible implementation of the first aspect, the data correction process includes at least one of slot replenishment, correction filtering, splitting multiple rounds of data, and diversity validation;

[0034] Slot filling includes filling empty slots with new data.

[0035] Correction filtering includes slot correction for dialogue data with abnormal slots, intent correction for dialogue data with abnormal intent recognition, or removal of abnormal dialogue data.

[0036] Splitting multi-turn data includes breaking down dialogue data containing multiple turns into single-turn dialogue data or two-turn dialogue data.

[0037] Diversity verification includes removing dialogue data whose sentence structure overlap exceeds a preset overlap value.

[0038] This application involves slot supplementation of dialogue data to provide effective data support when grouping data based on data type (e.g., slot inheritance), reducing the amount of invalid dialogue data. Dialogue data is corrected and filtered to correct or remove abnormal / erroneous dialogue data, improving its effectiveness. The first few rounds of multi-turn dialogues may bias and mislead model training, indirectly increasing the difficulty of model training. Splitting multi-turn dialogues can reduce the misleading impact of multi-turn data on model training. After verifying the diversity of dialogue data, the amount of dialogue data with high input content overlap (low effectiveness) in the original dataset can be appropriately reduced, while enriching the diversity of sentence structures (input content) with the same intent. This ensures that the dialogue data in the original dataset maintains input content diversity. Using a dataset with diverse input content for model training can improve the learning ability during the model training process, resulting in better model performance.

[0039] In another possible implementation of the first aspect, the method further includes:

[0040] The electronic device sorts the dialogue data in the training dataset according to the complexity of the dialogue data from low to high, thus obtaining the first training dataset.

[0041] The complexity of the dialogue data is used to characterize the accuracy of the large language model in predicting the intent of the dialogue data. The simpler the dialogue data, the higher the accuracy of the intent prediction and the lower the complexity of the dialogue data. The more complex the dialogue data, the lower the accuracy of the intent prediction and the higher the complexity of the dialogue data.

[0042] Electronic devices use training datasets as input to train large language models, obtaining the trained large language model, including:

[0043] The electronic device sequentially inputs the dialogue data from the first training dataset into the large language model for model training. When the model reaches convergence, the trained large language model is obtained.

[0044] This application introduces a course-based learning approach, where knowledge points are learned from easy to difficult. Dialogue data is sorted in order of learning difficulty from low to high. Based on the sorted dialogue data, a large language model is trained. This allows the large language model to learn and train on the dialogue data in the order of easy to difficult during the training process. This helps to make the training and learning of the large language model more efficient, thereby improving the training efficiency and the precision and recall of the large language model.

[0045] In another possible implementation of the first aspect, the electronic device sorts the dialogue data in the training dataset according to the complexity of the dialogue data from low to high, to obtain a first training dataset, including:

[0046] The electronic device divides the dialogue data in the training dataset into different data types, obtaining a subset of dialogue data corresponding to each data type.

[0047] The electronic device sorts the dialogue data in each subset of dialogue data according to the complexity of the dialogue data from low to high, and uses the combination of all sorted dialogue data subsets as the first training dataset.

[0048] In this application, dialogue data of each data type can be sorted to improve the efficiency of sorting and make the data distribution more orderly.

[0049] In another possible implementation of the first aspect, the electronic device sorts the dialogue data in each subset of dialogue data according to their complexity from low to high, and uses the union of all sorted subsets of dialogue data as the first training dataset, including:

[0050] For each data type of dialogue data subset, the electronic device sorts the dialogue data in the subset based on the number of dialogue turns in ascending order, and obtains the first sorted dialogue data subset.

[0051] Among them, the fewer the number of dialogue turns, the simpler and less complex the dialogue data; the more the number of dialogue turns, the more complex and more complex the dialogue data.

[0052] The electronic device sorts the dialogue data in the first subset of dialogue data based on the number of non-empty slots in the last round of dialogue in ascending order, and then obtains the sorted second subset of dialogue data.

[0053] Among them, the smaller the number of non-empty slots in the last round of dialogue data, the simpler the dialogue data and the lower its complexity; the larger the number of non-empty slots in the last round of dialogue data, the more complex the dialogue data and the higher its complexity.

[0054] The collection of the second dialogue data subsets corresponding to all data types is used as the first training dataset.

[0055] In this application, a larger number of dialogue turns indicates more complex dialogue data. Therefore, sorting the dialogue data from easiest to hardest based on the number of dialogue turns achieves the effect of sorting the dialogue data from easiest to hardest. Training a large language model based on the sorted dialogue data allows the large language model to be trained using the idea of ​​learning from easy to difficult in a curriculum, which can improve the training efficiency of the large language model. When the number of dialogue turns includes multiple turns, the dialogues in the preceding turns are subordinate dialogues and are in a completed state; moreover, during the training process of the large language model, intent recognition (intent distribution and slot extraction) is only performed on the last turn of dialogue, so the focus can be on the number of non-empty slots in the last turn of dialogue. The more non-empty slots there are, the more needs the input content of the dialogue data wants to express, and the more complex the dialogue data is. Therefore, sorting the dialogue data from easiest to hardest based on the number of non-empty slots achieves the effect of sorting the dialogue data from easiest to hardest. Training a large language model based on the sorted dialogue data allows the large language model to be trained using the idea of ​​learning from easiest to hardest in a curriculum, which can improve the training efficiency of the large language model.

[0056] In another possible implementation of the first aspect, the electronic device sorts the dialogue data in each subset of dialogue data according to their complexity from low to high, and uses the union of all sorted subsets of dialogue data as the first training dataset, including:

[0057] Electronic devices acquire quantized values ​​of dialogue data in a dataset to be sorted; quantized values ​​refer to the quantified values ​​of the accuracy of the large language model's predictions of dialogue data during the training process of the large language model.

[0058] The electronic device sorts the dialogue data in the dataset to be sorted according to the quantization value of each dialogue data from low to high, and obtains the third subset of dialogue data after sorting.

[0059] The first training dataset is the collection of the third dialogue data subsets corresponding to all data types.

[0060] The dataset to be sorted includes each subset of dialogue data, or each subset of second dialogue data.

[0061] In this application, a lower quantization value of the dialogue data indicates lower learning difficulty and a stronger learning / prediction ability of the large language model for that dialogue data; conversely, a higher quantization value indicates higher learning difficulty and a weaker learning / prediction ability of the large language model for that dialogue data. Sorting the dialogue data from low to high quantization values ​​achieves the effect of ranking the dialogue data from easy to difficult. Training the large language model based on the ranked dialogue data allows the large language model to be trained using a learning-from-easy-to-difficult approach, thus improving the training efficiency of the large language model.

[0062] In another possible implementation of the first aspect, the electronic device acquires quantized values ​​of the dialogue data in the second subset of dialogue data, including:

[0063] The electronic device divides the second dialogue data subset into a first quantization group and a second quantization group;

[0064] The first quantization group is used as training data, and the second quantization group is used as test data. The intent prediction results of each dialogue data in the second quantization group are obtained through the first preset model.

[0065] Based on the similarity between the intent prediction results of each dialogue data in the second quantization group and the actual intent results of each dialogue data in the second quantization group, the quantization value of each dialogue data in the second quantization group is obtained.

[0066] The second quantization group is used as training data, and the first quantization group is used as test data. The intent prediction results of each dialogue data in the first quantization group are obtained through the second preset model.

[0067] Based on the similarity between the intent prediction results of each dialogue data in the first quantization group and the actual intent results of each dialogue data in the first quantization group, the quantization value of each dialogue data in the first quantization group is obtained.

[0068] In this application, the first and second preset models can also be large language models. The two preset models are used to calculate the quantization values ​​of the dialogue data in the first and second quantization groups, respectively. The quantization value is calculated by combining the intent prediction result output by the model with the actual intent result indicated by the labels on the dialogue data itself. This quantization value better characterizes the learning difficulty / prediction difficulty of the large language model for the dialogue data, and the calculated quantization value is more accurate.

[0069] In another possible implementation of the first aspect, the electronic device sequentially inputs dialogue data from the first training dataset into the large language model for model training. When the model converges, the trained large language model is obtained, including:

[0070] The electronic device divides the first training dataset into data types to obtain dialogue datasets corresponding to each data type. The dialogue data in the dialogue datasets are ordered from lowest to highest complexity.

[0071] The electronic device divides the dialogue dataset corresponding to each data type into batch processing blocks in sequence to obtain multiple batch processing blocks filled with data.

[0072] The electronic device sequentially uses each batch processing block as input to train the large language model. When the model converges, the trained large language model is obtained.

[0073] In this application, due to the varying difficulty levels among dialogue data, directly inputting dialogue data into a large language model for training without differentiating between easy and difficult data may lead to the large language model over-training on simple data and under-training on complex data, resulting in poor training performance. In the above embodiments, by sorting and organizing the training dataset according to the first data dimension, the second data dimension, and the quantified value of (learning difficulty), the learning priority of each dialogue data in the training dataset is defined. Learning simple dialogue data first, followed by complex dialogue data, allows the large language model to be trained using a learning-from-easy-to-difficulty approach, thus improving training efficiency.

[0074] In another possible implementation of the first aspect, the electronic device trains the large language model by using the training dataset as input, and obtains the trained large language model, including:

[0075] The electronic device uses the training dataset as input to the large language model for the first stage of training. When the model reaches convergence, it obtains a candidate large language model.

[0076] The electronic device randomly samples a predetermined number of first dialogue data points from the training dataset. These first dialogue data points are single-intent dialogue data.

[0077] The electronic device acquires a preset amount of second dialogue data. This second dialogue data consists of dialogue data with two intents.

[0078] The electronic device uses the first dialogue data and the second dialogue data as input to the large language model for the second stage of training. When the model reaches convergence, the trained large language model is obtained.

[0079] This application introduces a two-stage fine-tuning method. In the first training stage, the large language model is fine-tuned using only single-intent dialogue data. In the second training stage, the large language model is trained using a mixture of single and dual-intent dialogue data. The trained target large language model performs better on the same test set, especially for intent recognition of dual-intent dialogue data, exhibiting higher accuracy and recall.

[0080] Secondly, a method for acquiring training data is provided, which is used to train a large language model, and the large language model is applied to human-computer dialogue scenarios.

[0081] The method includes:

[0082] The electronic device acquires the raw dataset. This raw dataset includes multiple dialogue data sets, which are used to characterize the data contained in dialogues within a human-computer dialogue scenario.

[0083] The electronic device categorizes the dialogue data in the original dataset into multiple groups based on preset data dimensions. Each group corresponds to one data dimension or a combination of data dimensions.

[0084] Electronic devices sample data from multiple groups, and use the combined sampled dialogue data as the training dataset for a large language model.

[0085] In this application, the electronic device analyzes the data dimensions and then processes the original dataset by grouping and performing within-group sampling. This ensures that the dialogue data in the constructed training dataset has a similar number of data points across multiple data dimensions, and that the dialogue data in the training dataset maintains a balanced distribution across these dimensions. Training a large language model using this training dataset results in a large language model that exhibits excellent performance across multiple data dimensions, demonstrating high accuracy and recall.

[0086] Thirdly, an intent recognition method is provided, which includes:

[0087] Electronic devices acquire user input statements in human-computer dialogue scenarios.

[0088] The electronic device inputs the input statement into a preset large language model and obtains the intent recognition result corresponding to the input statement. The intent recognition result includes the intent and slot corresponding to the input statement.

[0089] Based on the intent recognition results, the electronic device outputs the response content corresponding to the input statement.

[0090] The training method for the pre-defined large language model includes any of the large language model training methods provided in the first aspect above.

[0091] Specifically, the large language model is a model trained on a pre-defined training dataset. The process of obtaining the pre-defined training dataset includes: the electronic device dividing the dialogue data in the original dataset into multiple groups based on pre-defined data dimensions; the electronic device sampling data from each of these groups, and using the combined sampled dialogue data as the pre-defined training dataset. Each group corresponds to one data dimension or a combination of data dimensions.

[0092] In this application, the large language model is trained on a processed training dataset. This large language model exhibits excellent performance across multiple data dimensions, demonstrating high accuracy and recall. Therefore, the large language model trained on this dataset demonstrates superior performance across multiple data dimensions, exhibiting high accuracy and recall. Using this large language model for intent recognition in human-computer scenarios yields more accurate intent recognition results. For example, the large language model provides more accurate intent distribution and more accurate slot extraction / slot recall.

[0093] Fourthly, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0094] Fifthly, a computer-readable storage medium is provided that stores instructions which, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0095] In a sixth aspect, a computer program product comprising instructions is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0096] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to perform the method as described in any one of the first aspects.

[0097] Understandably, the beneficial effects that the electronic device described in the fourth aspect, the computer-readable storage medium described in the fifth aspect, the computer program product described in the sixth aspect, and the chip described in the seventh aspect can be achieved by referring to the beneficial effects of the first aspect and any possible design, or the second aspect and any possible design, or the third aspect and any possible design, which will not be repeated here. Attached Figure Description

[0098] Figure 1 A schematic diagram of the interface of a smart assistant for a mobile phone provided in an embodiment of this application;

[0099] Figure 2 A schematic diagram of a human-computer dialogue scenario for querying the weather provided in an embodiment of this application;

[0100] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0101] Figure 4 A schematic diagram illustrating the process of obtaining the training dataset in a model training method provided in this application embodiment;

[0102] Figure 5 A schematic diagram illustrating the process of obtaining the training dataset in another model training method provided in this application embodiment;

[0103] Figure 6 This application provides a schematic diagram of a sorting and organizing process based on a first data dimension.

[0104] Figure 7 This is a schematic diagram illustrating sorting and organizing based on a second data dimension, provided as an embodiment of this application.

[0105] Figure 8 This is a schematic diagram illustrating sorting based on quantized values, provided as an embodiment of this application.

[0106] Figure 9 This is a schematic diagram illustrating the model training process in a model training method provided in an embodiment of this application;

[0107] Figure 10 This application provides a schematic diagram of a complete model training method.

[0108] Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0109] Figure 12 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0110] In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0111] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0112] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0113] The development of artificial intelligence (AI) has been widely applied in dialogue scenarios, enabling natural language interaction between machines and humans. For example, large AI models (also known as large language models or large models) deployed on terminals can facilitate human-computer dialogue in various chat application scenarios provided by the terminal. Large language models refer to machine learning models with extremely large parameters (typically over a billion) and complex computational structures, capable of processing massive amounts of data and completing various complex tasks, such as natural language processing and image recognition. Large language models can be deployed on terminals or servers. The performance of the large language model determines the accuracy of the human-computer dialogue response.

[0114] Before introducing the model training method provided in the embodiments of this application, let's briefly introduce the human-computer dialogue scenario of intention recognition of dialogue commands based on a large language model.

[0115] Taking a mobile phone as an example, the phone can send dialogue commands in a human-computer dialogue scenario to a large language model. The large language model performs intent recognition on the dialogue command (such as intent distribution and slot extraction) and obtains the intent recognition result corresponding to the dialogue command. The large language model can return the intent recognition result corresponding to the dialogue command to the phone, and the phone can display the intent recognition result on the human-computer dialogue interface corresponding to the dialogue scenario. Alternatively, the large language model can call the corresponding agent to respond to the dialogue command based on the intent recognition result, obtain the output result corresponding to the dialogue command from the agent, and the phone can display the output result on the human-computer dialogue interface. The large language model can be deployed on the mobile phone or in the cloud.

[0116] In some embodiments, the human-computer dialogue scenario can be a human-computer dialogue scenario provided by a system service or application of the mobile phone. For example, the human-computer dialogue scenario can be a dialogue scenario provided by a smart assistant on the mobile phone. The smart assistant can be an application on the mobile phone, or it can be an application-layer service provided by the mobile phone.

[0117] For example, Figure 1 A schematic diagram of the interface of a mobile phone's smart assistant is provided. Figure 1 The interface 100 shown in (a) can be the main interface of the smart assistant. The main interface can be the default interface displayed after the phone starts the smart assistant, or it can be a recommended interface of the smart assistant (such as...). Figure 1 (as shown in (a)). Figure 1The main interface shown in (a) may include a sliding component 101 for providing hotspot information, a component 102 for providing smart functions, and a component 103 located at the bottom of the interface 100. For example, as Figure 1 The sliding component 101 shown in (a) may include a sliding component for "playing the movie / TV series 'xxx'" and a sliding component for "playing the song 'xxx'", etc. The mobile phone can trigger the execution of the corresponding operation when it receives a user's operation on the sliding component. For example, as... Figure 1 Component 102 shown in (a) may include calorie recognition, object recognition, QR code scanning, photo translation, etc. When a mobile phone receives a user's operation on component 102, it can trigger the corresponding operation to be performed. For example, ... Figure 1 Component 103 shown in (a) may include a "Recommended" component, a "Discover" component, a "Skills" component, and a "My" component for switching interfaces. The phone displays the corresponding recommended interface in response to the user's operation on the "Recommended" component; the phone displays the corresponding discovery interface in response to the user's operation on the "Discover" component; the phone displays the corresponding skills interface in response to the user's operation on the "Skills" component; and the phone displays the corresponding "My" interface in response to the user's operation on the "My" component. Component 103 also includes a wake-up component 104. When the phone receives an operation from the user on the wake-up component 104, it triggers the display of the corresponding interface. Specifically, the corresponding operation for the wake-up component 104 can be set according to the different input methods of the user in the human-computer dialogue scenario, such as text input or voice input.

[0118] For example, when the phone receives the user's first operation on the wake-up component 104, it triggers the display as shown below. Figure 1 The interface 200 shown in (b) is as follows. The first operation can be a single click, double click, long press, etc. Figure 1 As shown in (b), the voice dialogue interface can be an interface 200 displayed as a transparent floating window in the main interface 100 of the smart assistant. The interface 200 includes a keyboard component 201 and a scanning component 202.

[0119] When the mobile phone receives a user's operation on the keyboard component 201, the mobile phone can display as follows: Figure 1 Interface 203 is shown in (c). Figure 1As shown in (c), interface 203 can be a floating window displayed within the main interface 100 of the smart assistant. Interface 203 can be a dialogue interface for text input. Interface 203 may include a dialogue input box 204. The mobile phone can receive user input in the dialogue input box 204 and perform corresponding processing based on the input. Interface 203 may also include a voice conversion component 205. When the mobile phone receives user input on the voice conversion component 205, it displays a dialogue interface for voice input.

[0120] If the phone receives a user's operation on the voice conversion component 205 in interface 203, or if the phone receives a second operation from the user on the wake-up component 104 in interface 100, the phone can trigger the display as shown below. Figure 1 The voice dialogue interface is shown in (d). The second operation can be a long press, a single click, a double click, etc., and the first and second operations are different. For example... Figure 1 As shown in (d), the voice dialogue interface can be an interface 300 displayed as a floating window within the main interface 100 of the smart assistant. During user voice input, the phone can perform text recognition on the user's voice input and display the recognized text content in real-time in the input area 302 of the interface 300. Furthermore, to distinguish between the user's inputting state and paused input, the interface 300 also includes a wake-up component 301 with dynamic effects to indicate that the user is currently inputting. Optionally, the interface 300 may also include some quick phrase sliding components 303 to provide users with access to convenient functions.

[0121] In other scenarios, for example, users can also wake up the smart assistant through voice input in interface 100. When the phone detects that the smart assistant has been woken up, it displays the voice dialogue interface 300.

[0122] Understandably, a mobile phone can receive input actions triggered by the user on any display interface, acquire the user's input content, and perform corresponding interactive operations based on the input content. For example, the phone can receive the user's action of selecting a single component of a skill on the skills interface, display the corresponding dialogue interface, and acquire the user's input content. Alternatively, the phone can receive the user's first or second action on the wake-up component on the main interface, and display the corresponding dialogue interface. Or, the phone can respond to the user's voice activation of the smart assistant, and display the corresponding dialogue interface. In the above examples, the phone can acquire the user's input content through text or voice.

[0123] Let's take a user querying the weather as an example in a human-computer dialogue scenario. For instance, Figure 2 A schematic diagram of a human-computer dialogue scenario for querying the weather is provided.

[0124] like Figure 2 As shown in (a), the user input obtained by the mobile phone in the voice dialogue interface 300 is "Weather in City A". The mobile phone, based on the large language model built into the smart assistant (or the large language model built into the mobile phone), performs intent distribution and slot extraction on the input content to determine the intent recognition result corresponding to the input content. The intent recognition result may include the domain of the input content as "weather", the intent as "query the weather", and the slot-value as "region: City A, date: August 13th". The mobile phone can call the corresponding weather agent through a preset interface to obtain the output result corresponding to the intent recognition result. The mobile phone can display the output result in the dialogue interface 400. For example... Figure 2 As shown in (b), the output can include weather query results in text description format (such as...). Figure 2 (b) text box 402) and component-formatted weather query results (such as Figure 2 (b) Weather component 403). Figure 2 As shown in (b), the dialog interface 400 may also include user input content 401.

[0125] The above provides an example of a human-computer dialogue scenario. In real-world dialogue scenarios, human-computer dialogue may involve multi-turn conversations. These multi-turn dialogue commands typically contain multiple intents and slots. Therefore, a large language model is needed to accurately distribute the intents within the user's dialogue commands and extract the relevant slot content to obtain a relatively accurate intent recognition result. Furthermore, based on the intent recognition result, the model can invoke the corresponding agent to respond, or output the corresponding result. Therefore, how to train a high-performance large language model becomes a crucial issue.

[0126] A common model training method is to randomly sample the original dataset, or to sample the original dataset for each intent, to obtain a training dataset. The large language model is then trained based on this training dataset, and a well-trained large language model is obtained when the model reaches preset performance metrics.

[0127] However, dialogue data in different human-computer dialogue scenarios, and even within the same type of dialogue scenario, may contain multi-dimensional differences in intent and slots. For example, multiple dialogue data may differ in intent, slots, number of dialogue turns, and data type. Furthermore, dialogue data can be categorized into single-intent dialogue data and multi-intent dialogue data based on the number of intents. Ignoring the multi-dimensional differences in dialogue data—that is, not considering the impact of the diversity and balance of dialogue data across multiple dimensions on model training—directly randomly sampling the original dataset to obtain the training dataset may result in a large language model trained on the training dataset that suffers from poor performance, low accuracy, and low recall. Moreover, training a model based on randomly sampled dialogue data (the training dataset) also suffers from low training efficiency.

[0128] When large language models with poor performance are applied to human-computer dialogue scenarios, they may encounter problems such as inaccurate or unrecognizable recognition of dialogue commands. Therefore, it is crucial to train a large language model that performs well across all intents, slots, dialogue turns, and data types.

[0129] This application provides a model training method applicable to the training of large language models. Based on a preset grouping and sampling strategy, the original dataset is grouped and sampled across multiple data dimensions to obtain a training dataset. The training dataset maintains a balanced distribution across multiple data dimensions. Training the large language model using this training dataset results in a model that exhibits excellent performance across multiple data dimensions, demonstrating high accuracy and recall.

[0130] Furthermore, the dialogue data in the training dataset can be sorted and arranged in order of increasing difficulty, and then trained on the large language model in that order. This allows the large language model to learn and train in accordance with the principle of increasing difficulty, which not only ensures the training effect of the large language model but also improves its training efficiency.

[0131] The model training method provided in this application can be applied to electronic devices. These electronic devices can be portable computers (such as mobile phones), tablets, laptops, personal computers (PCs), servers (such as cloud servers, dedicated servers, server clusters, etc.), etc. The following embodiments do not impose any special limitations on the specific form of the electronic device.

[0132] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0133] See Figure 3, Figure 3 The illustrated electronic device 100 may include a processor 101, a memory 102, a communication interface 103, and a bus 104. The processor 101, the memory 102, and the communication interface 103 can be connected via the bus 104.

[0134] The processor 101 is the control center of the electronic device. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0135] As an example, processor 101 may include one or more CPUs, for example Figure 3 CPU 0 and CPU 1 are shown in the diagram.

[0136] In this embodiment, the electronic device may have a built-in large language model. The processor 101 can input dialogue commands into the large language model for intent recognition (intent distribution and slot extraction) to obtain the intent recognition result of the dialogue commands. The processor 101 can also group, sample, and quantize the original dataset across multiple data dimensions based on a preset grouping and quantization strategy to obtain a training dataset. The training dataset is then input into the large language model for model training to obtain a trained large language model, which is used for intent recognition of dialogue commands in human-computer dialogue scenarios.

[0137] The memory 102 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0138] In some embodiments, the memory 102 may store the original dataset and the training dataset. A database may be deployed in the memory 102 to store the original dataset and the training dataset.

[0139] In one possible implementation, the memory 102 may exist independently of the processor 101. The memory 102 can be connected to the processor 101 via a bus 104 and is used to store data, instructions, or program code. When the processor 101 calls and executes the instructions or program code stored in the memory 102, it can implement the split-screen display method provided in this application embodiment.

[0140] In another possible implementation, the memory 102 can also be integrated with the processor 101.

[0141] Communication interface 103 is used for the electronic device to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Communication interface 203 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0142] When the electronic device acts as a server, it can communicate with other devices that need to access the large language model via communication interface 103 to respond to human-computer dialogues on those devices. For example, the electronic device can receive query requests carrying dialogue commands from other devices through the communication interface, input the dialogue commands into the large language model, and obtain the corresponding intent recognition results. The electronic device can then return the intent recognition results to the other devices.

[0143] Bus 104 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0144] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the electronic device, except... Figure 3 In addition to the components shown, the electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0145] The model training method provided in this application includes a training dataset acquisition stage and a model training stage.

[0146] Figure 4 A schematic diagram illustrating the process of obtaining the training dataset in a model training method is provided, for reference. Figure 4 ,include:

[0147] S201. Electronic devices acquire the raw dataset.

[0148] The original dataset can come from dialogue datasets in open-source databases; or it can be a dialogue dataset generated by a pre-defined model / algorithm.

[0149] The original dataset includes multiple dialogue data points. Each dialogue data point may include at least the user's input, the intent and slot obtained after intent recognition based on the input, the corresponding response content, and the data type.

[0150] Here, data types refer to several typical data types summarized in this embodiment based on the analysis and classification of a large amount of dialogue data. For example, based on the number of dialogue turns, they can be divided into single-turn dialogues and multi-turn dialogues. In multi-turn dialogues, according to the intent and slots of the multi-turn dialogue, data types can be further divided into different types such as follow-up questions, domain switching, secondary execution, intent inheritance, and slot inheritance.

[0151] The following examples illustrate dialogue data and its data types.

[0152] For example, a dialogue data 1 can be: {"dialog":[{"user":"Share location with Wang using application 1","state":"Share location information (application name = application 1, contact = Wang)","assistant":"Okay, sharing successful, the other party has confirmed"},"type":"single round"}.

[0153] In dialogue data 1, the user's input is "Share location with Wang using application 1"; the intent obtained after intent recognition based on the input is "Share location information"; the slots include "Application name = application 1, contact = Wang"; the corresponding response is "Okay, sharing successful, the other party has confirmed"; the data type is single round.

[0154] A single turn refers to a dialogue data set containing only one round of conversation; that is, the user provides one input, and the large language model provides one output. Correspondingly, a dialogue that includes at least two rounds is called a multi-turn dialogue. Multi-turn dialogues can be further subdivided into different data types.

[0155] For example, data types for multi-turn dialogues can also include slot inheritance.

[0156] For example, a dialogue data 2 can be: {"dialog":[{"user":"Share location with Wang using application 1","state":"Share location information (application name = application 1, contact = Wang)","assistant":"Okay, sharing successful, the other party has confirmed"},{"user":"Help me make a voice call to her","state":"Start social application call (contact = Wang, call mode = voice call, application name = None)","assistant":null}],"type":"AB slot inheritance"}.

[0157] In the first round of dialogue data 2, the user's input was "Share my location with Wang using application 1"; the intent obtained after intent recognition based on the input was "Share location information"; the slots included "Application Name = Application 1, Contact = Wang"; the corresponding response was "Okay, sharing successful, the other party has confirmed". In the second round of dialogue, the user's input was "Make a voice call for her"; the intent obtained after intent recognition based on the input was "Initiate a call using a social application"; the slots included "Contact = Wang, Call Mode = Voice Call, Application Name = None"; the corresponding response was "null"; the data type was slot inheritance.

[0158] Dialogue data of data type slot inheritance, with at least two dialogue rounds. Slot inheritance means that at least one slot in the second round of dialogue is the same as a slot in the first round of dialogue, and that slot is null in the second round of dialogue. In this case, it is necessary to inherit the value of that slot from the first round of dialogue and perform corresponding processing. The first round of dialogue is the round preceding the second round of dialogue.

[0159] The first round of input in the aforementioned dialogue data 2 included "Application 1," and the corresponding slot for the first round of dialogue included "Application Name = Application 1." The second round of input did not include the application name, and the corresponding slot for the second round of dialogue contained "Application Name = None." Since the application name cannot be determined solely from the slots in the second round of dialogue, the intent to "initiate a social application call" cannot be executed. In this case, slot inheritance can be used to obtain the slot "Application Name = Application 1" from the first round of dialogue. The "Application Name" in the second round of dialogue inherits from the "Application Name" in the first round of dialogue, thus confirming the application name as Application 1. Through slot inheritance, the intent to "initiate a social application call" can be executed, allowing the operation of initiating Application 1 to conduct a voice call with Wang to be performed.

[0160] For example, data types for multi-turn dialogues can also include intent inheritance.

[0161] For example, a dialog data 3 could be: {"dialog":[{"user":"Help me create an alarm clock to wake me up at 8 am","state":"Create alarm clock(event=wake up, time=8 am, date=None)","assistant":"Okay, created successfully"},{"user":"I don't want it anymore, please cancel it","state":"Cancel alarm clock(time=8 am, event=wake up, range type=None)","assistant":null}],"type":"AB intent inheritance"}.

[0162] In the first round of dialogue data 3, the user's input was "Help me set an alarm to wake me up at 8 AM." Intent recognition based on this input yielded the intent "Create Alarm." The slots included "Event = Wake Up, Time = 8 AM, Date = None." The corresponding response was "Okay, created successfully." In the second round of dialogue, the user's input was "I don't want it anymore, please cancel it." Intent recognition based solely on the second round of dialogue wouldn't capture the intent for that round. In this case, intent inheritance can be used to obtain the intent "Create Alarm" from the first round, thus determining the intent for the second round as "Cancel Alarm." Based on the second round's input and intent inheritance, the resulting intent is "Cancel Alarm." The slots included "Time = 8 AM, Event = Wake Up, Range Type = None." The corresponding response was "null," and the data type was intent inheritance.

[0163] The data type is dialog data with intent inheritance, and its dialogue rounds include at least two rounds. Intent inheritance means that the intent of the second round of dialogue inherits the intent of the first round of dialogue. The first round of dialogue is the round preceding the second round of dialogue.

[0164] In the dialogue data 3 above, the input in the first round is "Help me set an alarm clock to wake me up at 8 am," and the intent of the first round of dialogue is "create alarm clock." The input in the second round does not contain any intent. Intent recognition cannot be performed based solely on the second round of dialogue, so the response "I don't want it anymore, please cancel it" cannot be executed. In this case, intent inheritance can be used to obtain the intent "create alarm clock" from the first round of dialogue, and determine that the object corresponding to the intent "cancel" is "alarm clock." Through intent inheritance, the intent of the input "I don't want it anymore, please cancel it" can be determined as "cancel alarm clock," and the operation of canceling the alarm clock can be executed.

[0165] For example, the data types of multi-turn dialogues can also include follow-up questions.

[0166] For example, a dialogue data 4 could be: {"dialog":[{"user":"What's the weather like in City A today?","state":"Check the weather (location = City A, date = today (20240925))","assistant":"The weather in City A is sunny today, with a high of 28℃ and a low of 21℃"},{"user":"Check the temperature in City A at 8 PM tonight","state":"Check the weather (location = City A, date = today (20240925), time = 8 PM)","assistant":"The temperature in City A at 8 PM tonight is 23℃}"],"type":"Follow-up question"}.

[0167] In the first round of dialogue data 4, the user's input was "What's the weather like in City A today?" Intent recognition based on the input revealed the intent as "Check the weather." The slot included "Location = City A, Date = Today (20240925)". The corresponding response was "The weather in City A is sunny today, with a high of 28℃ and a low of 21℃." In the second round of dialogue, the user's input was "Check the temperature in City A at 8 PM tonight." This was a follow-up question, asking for a more specific temperature at a particular time in City A today, building upon the first round. The intent in the second round was still "Check the weather." The slot included "Location = City A, Date = Today (20240925), Time = 8 PM". The corresponding response was "The temperature in City A at 8 PM tonight is 23℃." This type of multi-turn dialogue data is a follow-up question.

[0168] For example, the data types for multi-turn dialogues can also include field cuts.

[0169] For example, a dialogue data 5 could be: {"dialog":[{"user":"What's the weather like in City A today?","state":"Check the weather (location = City A, date = today (20240925))","assistant":"The weather in City A is sunny today, with a high of 28℃ and a low of 21℃"},{"user":"Check flights from City A to City B today","state":"Check flights (departure point = City A, destination = City B, date = today (20240925))","assistant":"Flights from City A to City B today include: [flight card]}]","type":"field"}. The flight card could include flight information from City A to City B after the query time, such as September 25, 2024.

[0170] In the first round of dialogue data 5, the user's input was "What's the weather like in City A today?"; the intent identified after intent recognition was "Check the weather"; the slots included "Location = City A, Date = Today (20240925)"; the corresponding response was "The weather in City A is sunny today, with a high of 28℃ and a low of 21℃." In the second round of dialogue, the user's input was "Check flights from City A to City B today," and the intent was "Check flights"; the slots included "Departure point = City A, Destination = City B, Date = Today (20240925)"; the corresponding response was "Flights from City A to City B today include: [Flight Card]." The intents in the second round of dialogue are completely different from those in the first round. The domain for "Check the weather" is "weather," allowing the use of a weather agent to query weather information; the domain for "Check flights" is "flights," allowing the use of a travel agent to query flight information. A multi-turn dialogue in a single dialogue dataset involves domain switching; the data type of this multi-turn dialogue is called a domain switching.

[0171] For example, the data types of multi-turn dialogues can also include secondary execution.

[0172] For example, a dialogue data 6 could be: {"dialog":[{"user":"Help me create an alarm clock to wake me up at 8 AM","state":"Create alarm clock(event=wake up, time=8 AM, date=None)","assistant":"Okay, created successfully"},{"user":"Create another alarm clock at 8:20 AM","state":"Create alarm clock(time=8:20 AM, event=wake up, date=None)","assistant":"Okay, created successfully"}],"type":"Second execution"}.

[0173] In the first round of dialogue data 6, the user's input was "Create an alarm clock to wake me up at 8 AM." Intent recognition based on the input revealed the intent as "Create alarm clock." The slots included "Event = Wake up, Time = 8 AM, Date = None." The corresponding response was "Okay, created successfully." In the second round of dialogue, the user's input was "Create another alarm clock for 8:20 AM." The intent in the second round was still "Create alarm clock." The slots included "Time = 8:20 AM, Event = Wake up, Date = None." The corresponding response was "Okay, created successfully." This type of multi-round dialogue with the same intent and slot type in both rounds is considered a secondary execution.

[0174] The above embodiments illustrate typical data types for dialogue data. It is understood that, in addition to the data types listed above, dialogue data can also include other types. This embodiment only uses the above typical data types as examples to process the original dataset; no specific data type is limited.

[0175] In some embodiments, multiple dialogue data in the original dataset may have some data problems. In this embodiment, multiple data in the original dataset can also be corrected.

[0176] For example, data correction processing may include slot filling. Slot filling refers to supplementing or correcting slots in dialogue data that are empty or have abnormal slots. For instance, the input content of a dialogue is "Help me set an alarm clock to wake me up at 8 am"; the intent obtained after intent recognition based on the input content is "Create alarm clock"; the slot is empty. Slots can be extracted from this dialogue data, but due to various anomalies, the slots are empty. In this case, slot filling can be performed on the dialogue data, adding the slot "Event = Wake up, Time = 8 am, Date = None" to make the dialogue data more complete and effective. For example, the input content of a dialogue is "Tell a joke"; the intent obtained after intent recognition based on the input content is "Write a joke"; the slot is empty. The slots in this dialogue data are inherently empty. In this case, slot filling can also be performed on the dialogue data, adding the slot "Event = Write a joke" to make the dialogue data more complete and effective.

[0177] Slotting dialogue data can provide effective data support when grouping data based on data type (e.g., slot inheritance), reducing the amount of invalid dialogue data.

[0178] In this embodiment, when the original dataset contains a sufficient amount of dialogue data (e.g., when the number of dialogue data exceeds a first value), dialogue data with empty slots can be directly discarded. When the original dataset contains a limited amount of dialogue data (e.g., when the number of dialogue data is less than a second value), empty dialogue data can be added to fill the slots, and the specific method can be determined based on the actual situation. The first value is greater than the second value.

[0179] For example, data correction processing can also include correction filtering. Correction filtering refers to correcting or removing some abnormal / erroneous dialogue data. For instance, the input content of a dialogue data is "Query the weather in City A today"; after intent recognition based on the input content, the intent obtained is "Query the weather"; the slots include "Location = City B, Date = Today (20240925)". The input content of the dialogue data includes City A but not City B, so the dialogue data is erroneous. For such dialogue data, slot correction can be performed, changing the value corresponding to "Location" from "City B" to "City A", resulting in the slot "Location = City A", making the dialogue data correct and valid. Alternatively, since the original dataset contains a large amount of dialogue data, this dialogue data can also be directly discarded, making the dialogue data in the original dataset more valid.

[0180] In this embodiment, when the original dataset contains a sufficient amount of dialogue data, for example, when the number of dialogue data is greater than a first value, dialogue data with abnormalities / errors can be directly discarded; when the original dataset contains a small amount of dialogue data, for example, when the number of dialogue data is less than a second value, dialogue data with abnormalities / errors can be corrected. The specific method of correction and filtering can be determined according to the actual situation.

[0181] For example, data correction processing may also include splitting multi-turn dialogue data. Splitting multi-turn dialogue data refers to dividing dialogue data containing multiple turns into segments based on the number of turns. Here, multi-turn dialogue specifically refers to dialogues with three or more turns, such as four-turn, five-turn, or six-turn dialogues. In human-computer dialogue scenarios, the most valuable information from multi-turn dialogues often lies in the final turn. Therefore, the earlier turns of a multi-turn dialogue may bias and mislead the model during training, indirectly increasing the difficulty of model training.

[0182] For multi-turn dialogues, they can be broken down into single-turn dialogues. For example, the last turn can be kept as a new dialogue data entry; or the first turn can be kept as a new dialogue data entry; or any turn can be kept as a new dialogue data entry. The data type of the new dialogue data entry is a single turn. Alternatively, the dialogue data of the multi-turn dialogue can be discarded from the original dataset. Alternatively, the multi-turn dialogue can be directly split into multiple new two-turn dialogue data entries using the same method as single-turn dialogues.

[0183] Alternatively, for such multi-turn dialogues, they can be split into two-turn dialogues. For example, the last two turns can be retained as a new dialogue data entry; or the first two turns can be retained as a new dialogue data entry; or any two turns can be retained as a new dialogue data entry. The data type of the new dialogue data is determined based on the intent and slots of the two-turn dialogues. Furthermore, the dialogue data of this multi-turn dialogue can be discarded from the original dataset. Alternatively, the multi-turn dialogue can be directly split into multiple new two-turn dialogue data entries using a two-turn dialogue splitting method.

[0184] Alternatively, a multi-turn dialogue can be split into three turns. Similarly, the last three turns can be retained as a new dialogue data entry; or, the first three turns can be retained as a new dialogue data entry; or, any three turns can be retained as a new dialogue data entry. The data type of this new dialogue data is determined based on the intent and slots of the three turns. Furthermore, the dialogue data from this multi-turn dialogue can be discarded from the original dataset.

[0185] This embodiment does not limit the specific number of rounds to be split in a multi-turn dialogue. The purpose of splitting the multi-turn dialogue data is to improve the effectiveness of the dialogue data. The specific number of rounds to be split can be determined according to the actual situation.

[0186] For example, the data correction process may also include diversity verification of dialogue data. Dialogue data diversity verification refers to filtering dialogue data based on its intent and slot, focusing on input content (dialogue phrase / statement / query) that has a high degree of overlap (e.g., an overlap higher than a preset overlap value). For instance, the intent of dialogue data 1 is "check the weather," the intent of dialogue data 2 is also "check the weather," and the intent of dialogue data 3 is also "check the weather." Specifically, the input content (phrase) of dialogue data 1 is "check the weather in City A today," the input content (phrase) of dialogue data 2 is "check the weather in City B today," and the input content of dialogue data 3 is "what clothes are suitable to wear in City A today."

[0187] Given three dialogue data sets with the same intent, dialogue data 1 and dialogue data 2 have a very high degree of overlap in their input content. Such dialogue data with high input content overlap is of little significance for model training; therefore, highly overlapping dialogue data can be selectively discarded. This selective discarding can be based on dimensions such as the number of turns and the data type. For example, dialogue data 1 is of single-turn data type, while dialogue data 2 is of intent inheritance data type. Dialogue data 2 is more complex and more effective than dialogue data 1, so dialogue data 2 can be retained while dialogue data 1 can be discarded. This embodiment does not limit the strategy and principles of selective discarding; one or more highly overlapping dialogue data sets can be discarded based on the actual situation.

[0188] Although the three dialogue data have the same intent, the input content of dialogue data 3 has a low degree of overlap with dialogue data 1 / dialogue data 2. The input content of dialogue data 3 can increase the diversity of dialogue data under the same intent. Therefore, dialogue data 3 is retained.

[0189] After verifying the diversity of dialogue data, the amount of dialogue data with high overlap (low effectiveness) in the original dataset can be appropriately reduced, and the diversity of sentence patterns (input content) with the same intent can be enriched. This ensures that the dialogue data in the original dataset maintains the diversity of input content. Using a dataset with diverse input content for model training can improve the learning ability of the model during the training process, resulting in better performance of the trained model.

[0190] S202. The electronic device divides the original dataset into groups based on at least one preset data dimension, resulting in multiple groups corresponding to the original dataset.

[0191] The preset data dimensions can include data types, such as those mentioned in S201, which can include single-turn, follow-up question, domain-specific, secondary execution, intent inheritance, and slot inheritance. The original dataset is grouped based on different data types. For example, the dictionary container (dict1) corresponding to group 1 is ['single-turn'] = [dialogue data 1, ...], indicating that group 1, whose data type is single-turn, includes dialogue data 1, etc. Similarly, the dictionary container (dict2) corresponding to group 2 is ['follow-up question'] = [dialogue data 2, ...], indicating that group 2, whose data type is follow-up question, includes dialogue data 2, etc.

[0192] In some embodiments, the preset data dimension may further include intent classification. The original dataset is grouped according to different intent classifications. For example, the dictionary container (dict3) corresponding to group 3 is ['Check Weather'] = [Dialogue Data 3, ...], indicating that group 3, whose intent is to check the weather, includes dialogue data 3, etc. Similarly, the dictionary container (dict4) corresponding to group 4 is ['Send Text Message'] = [Dialogue Data 4, ...], indicating that group 4, whose intent is to send a text message, includes dialogue data 4, etc.

[0193] In some embodiments, the preset data dimension may also include the number of dialogue turns. The original dataset is grouped according to the number of dialogue turns. For example, the dictionary container (dict5) corresponding to group 5 is ['1 turn'] = [dialogue data 5, ...], indicating that group 5, with 1 dialogue turn, includes dialogue data 5, etc. Similarly, the dictionary container (dict6) corresponding to group 6 is ['2 turns'] = [dialogue data 6, ...], indicating that group 6, with 2 dialogue turns, includes dialogue data 6, etc.

[0194] In some embodiments, the preset data dimension may also include combinations of non-empty slots. The original dataset is grouped according to the different combinations of non-empty slots. For example, the dictionary container (dict7) corresponding to group 7 is ['location_time'] = [dialogue data 7, ...], indicating that group 7, which contains time and location in its non-empty slot combinations, includes dialogue data 7, etc. Similarly, the dictionary container (dict8) corresponding to group 8 is ['content_application name_contact'] = [dialogue data 8, ...], indicating that group 8, which contains content, application name, and contact in its non-empty slot combinations, includes dialogue data 8, etc.

[0195] The above embodiments all illustrate the grouping of dialogue data in the original dataset using a single preset data dimension. In some feasible methods, to reduce the number of groups and improve the effectiveness of the grouped data, multiple preset data dimensions can be combined to group the dialogue data in the original dataset. When grouping dialogue data in the original dataset using multiple preset data dimensions, the priority of each data dimension can be set. For example, in the example of grouping based on two data dimensions, data dimension 1 is set to the first priority, and data dimension 2 to the second priority, with the first priority higher than the second priority. During the grouping process, the data is first grouped by the higher-priority data dimension, and then by the lower-priority data dimension. For example, when data dimension 1 is data type and data dimension 2 is intent classification, the dialogue data in the original dataset is first grouped by different data types. For groups of the same data type, the dialogue data in that group is then grouped by intent classification.

[0196] For example, the resulting groupings could include: Group 9, whose dictionary container (dict9) is ['Single Turn']['Check Weather'] = [Dialogue Data 9, ...], indicating that Group 9, whose data type is single turn and whose intent is to check the weather, includes dialogue data 9, etc. Similarly, Group 10, whose dictionary container (dict10) is ['Single Turn']['Send Text Message'] = [Dialogue Data 10, ...], indicates that Group 10, whose data type is single turn and whose intent is to send text messages, includes dialogue data 10, etc. Group 11, whose dictionary container (dict11) is ['Follow-up Question']['Check Weather'] = [Dialogue Data 11, ...], indicates that Group 11, whose data type is follow-up question and whose intent is to check the weather, includes dialogue data 11, etc. Finally, Group 12, whose dictionary container (dict12) is ['Follow-up Question']['Send Text Message'] = [Dialogue Data 12, ...], indicates that Group 12, whose data type is follow-up question and whose intent is to send text messages, includes dialogue data 12, etc.

[0197] In this way, we can first group based on the data type "single round" or "follow-up question" (dict9∪dict10, or dict11∪dict12), and then group them based on the intent category "check the weather" or "send a text message" within the same data type, thus obtaining each group (dict9, dict10, dict11, dict12).

[0198] In some embodiments, when the preset data dimensions include data type, intent classification, number of dialogue turns, and non-empty slot combinations, the dialogue data in the original dataset can be classified based on these four preset data dimensions. Correspondingly, the priority of each data dimension can be set. In one example, data type can be set as the first priority, intent classification as the second priority, number of dialogue turns as the third priority, and non-empty slot combinations as the fourth priority. The first priority is higher than the second priority, the second priority is higher than the third priority, and the third priority is higher than the fourth priority. It is understood that the priority of different preset data dimensions is not limited; the priority of each data dimension can be set according to the needs of the actual model and the scenario. For example, in scenarios requiring higher accuracy in intent recognition, intent classification can be set as the highest priority.

[0199] Let's take grouping the dialogue data in the original dataset using four preset data dimensions as an example. The priorities of the four data dimensions are the same as those in the example above.

[0200] Specifically, the dialogue data in the original dataset is first grouped by different data types. For dialogue data of the same data type, it is then grouped by intent classification. For dialogue data of the same intent classification, it is then grouped by the number of dialogue turns. Finally, for dialogue data of the same number of dialogue turns, it is grouped by non-empty slot combination.

[0201] For example, the resulting grouping may include: the dictionary container (dict13) corresponding to group 13 is ['single round']['check the weather']['1 round']['location_time'] = [dialogue data 13...], indicating that group 13, which has a data type of single round, an intent of checking the weather, a dialogue round of 1, and slots including location and time, includes dialogue data 13, etc.

[0202] The dictionary container (dict14) corresponding to group 14 is ['single round']['send SMS']['1 round']['content_application name_contact'] = [dialogue data14...], which means that group 14, which has a data type of single round, an intent of sending SMS, a dialogue round of 1, and slots including content, application name, and contact, includes dialogue data 14.

[0203] The dictionary container (dict15) corresponding to group 15 is ['single round']['send SMS']['1 round']['content_application name_contact_SMS type'] = [dialogue data15...], indicating that group 15, which has a data type of single round, an intent of sending SMS, a dialogue round of 1, and slots including content, application name, contact, and SMS type, includes dialogue data 15.

[0204] The dictionary container (dict16) corresponding to group 16 is ['Ask a follow-up question']['Check the weather']['2 rounds']['Location_Time'] = [Dialogue data 16...], which means that group 16, whose data type is follow-up question, whose intent is to check the weather, whose dialogue rounds are 2, and whose slots include location and time, includes dialogue data 16.

[0205] The dictionary container (dict17) corresponding to group 17 is ['Ask a follow-up question']['Send a text message']['2 rounds']['Content_Application name_Contact'] = [Dialogue data 17,...], which means that group 17, which has the data type "Ask a follow-up question", the intent "Send a text message", the number of dialogue rounds is 2, and the slots include content, application name and contact, includes dialogue data 17.

[0206] The dictionary container (dict18) corresponding to group 18 is ['Ask a follow-up question']['Send an SMS']['2 rounds']['Content_Application Name_Contact_SMS Type'] = [Dialogue Data 18...], which means that group 18, whose data type is "Ask a follow-up question", whose intent is "Send an SMS", whose dialogue rounds are 2, and whose slots include content, application name, contact, and SMS type, includes dialogue data 18.

[0207] In this embodiment, the electronic device divides the original dataset into groups based on at least one preset data dimension, resulting in n groups, which can be represented as: [d1,d2,d3,d4,d5,d6,…,dn].

[0208] S203. The electronic device samples the n groups corresponding to the original dataset and merges the sampled dialogue data to obtain the training dataset.

[0209] In this embodiment, the electronic device can obtain a preset number of dialogue data from each group, and merge the dialogue data obtained from the groups to obtain a training dataset. This training dataset is then input into an initial large language model for model training until the large language model converges or until the loss value of the large language model is less than a preset loss value, thus obtaining a trained large language model.

[0210] In one feasible embodiment, since the data types "single-turn," "follow-up question," and "cutting domain" have relatively low granularity, the number of groups corresponding to these data types is relatively large. Therefore, a first number of dialogue data can be sampled from each group corresponding to these data types. Since the data types "intent inheritance" or "slot inheritance" have higher granularity, the number of groups corresponding to these data types is less than the number of groups corresponding to these data types. Therefore, a second number of dialogue data can be sampled from each group corresponding to these data types. The second number is greater than the first number. For example, the first number can be 5, 6, 7, 8, etc.; the second number can be 10, 11, 12, 13, etc.

[0211] The electronic device samples a corresponding number of dialogue data from each group corresponding to each data type according to the sampling quantity corresponding to the grouping of different data types, and merges all the sampled dialogue data into a training dataset.

[0212] It is understandable that the determination of the first and second quantities also takes into account the balanced distribution of the training dataset across various data dimensions. That is, even if the first and second quantities are different, the relative balance of the training dataset across various data dimensions can still be achieved.

[0213] In this embodiment, the electronic device can group and sample the dialogue data in the original dataset based on at least one preset data dimension, which can ensure that the constructed training dataset maintains a balanced distribution across multiple data dimensions. Training the large language model based on the balanced training dataset can improve the efficiency of model training. In addition, data correction processing on the original dataset can ensure the validity of the dialogue data in the original dataset (which can also be the training dataset). Training the model based on the training dataset can improve the accuracy and recall of the large language model.

[0214] In some embodiments, a learning process is introduced, where knowledge points are learned from easy to difficult. Further data processing of the training dataset can also be performed. For example, specifically, for each type of dialogue data, the dialogue data of the same type are sorted in order of complexity / learning difficulty from low to high. Based on the sorted dialogue data, the large language model is trained. This allows the large language model to learn from the dialogue data in order of increasing difficulty during training, which helps to make the training of the large language model more efficient, thereby improving the training efficiency and the precision and recall of the large language model.

[0215] In one feasible implementation method Figure 5 This provides a schematic diagram illustrating the process of obtaining the training dataset in another model training method. (See reference...) Figure 5 After the electronic device obtains the training dataset after data correction, grouping, sampling, and merging based on the above S201-S203, it may further include:

[0216] S301. The electronic device divides the dialogue data in the training dataset into data types according to the data type, and obtains a subset of dialogue data corresponding to each data type.

[0217] In this embodiment, each piece of dialogue data includes a data type (i.e., a label indicating the data type). The electronic device can extract dialogue data belonging to the same data type as a subset of dialogue data.

[0218] For example, data types include such as Figure 5The diagram illustrates single-turn dialogue, follow-up questions, domain-based dialogue, secondary execution, intent inheritance, and slot inheritance. The electronic device extracts data of type "single-turn" from the training dataset, forming subset 1 of the dialogue data corresponding to "single-turn"; extracts data of type "follow-up questions," forming subset 2 of the dialogue data corresponding to "follow-up questions"; extracts data of type "domain-based dialogue," forming subset 3 of the dialogue data corresponding to "domain-based dialogue"; extracts data of type "secondary execution," forming subset 4 of the dialogue data corresponding to "secondary execution"; extracts data of type "intent inheritance," forming subset 5 of the dialogue data corresponding to "intent inheritance"; and extracts data of type "slot inheritance," forming subset 6 of the dialogue data corresponding to "slot inheritance."

[0219] After obtaining the dialogue data subsets corresponding to each data type, the electronic device can organize and sort the dialogue data in each subset. In this embodiment, the purpose of organizing and sorting the dialogue data is to rank it according to complexity. When used as a training dataset, the data is sequentially input into the large language model for training, placing the large language model in a learning environment from simple to complex. This improves the training efficiency and learning ability of the large language model, thereby increasing its accuracy and recall.

[0220] For each subset of dialogue data, sorting can be performed based on at least one preset data dimension. For example, sorting can be done by the number of dialogue turns and / or the number of non-empty slots. The learning difficulty of the dialogue data also affects the training of the large language model. In this embodiment, a quantified value of the learning difficulty of each dialogue data set can be obtained, and the dialogue data can be sorted based on the quantified value. Here, learning difficulty refers to the accuracy of the large language model in learning and predicting the dialogue data. The learning difficulty can also refer to the complexity of the dialogue data. The simpler the dialogue data, the higher the accuracy of the large language model in predicting the intent of the dialogue statement, and the lower the learning difficulty / prediction difficulty. Conversely, the more complex the dialogue data, the lower the accuracy of the large language model in predicting the intent of the dialogue statement, and the higher the learning difficulty / prediction difficulty. Regarding the quantified value of the dialogue data complexity, a lower quantified value indicates simpler dialogue data, and a higher quantified value indicates more complex dialogue data.

[0221] Dialogue data in different dialogue scenarios have varying degrees of learning difficulty. For example, dialogue data from low-frequency scenarios where user input contains a small number of intents will theoretically be easier to learn; while dialogue data from high-frequency scenarios where user input contains a large number of intents will theoretically be more difficult to learn.

[0222] In some feasible implementations, the electronic device organizes and sorts the dialogue data in the subset of dialogue data corresponding to each data type, including:

[0223] S302. The electronic device sorts the dialogue data in the dialogue data subset from the first data dimension to obtain the sorted first dialogue data subset.

[0224] For example, the first data dimension can be the number of dialogue turns. The electronic device can sort the dialogue data according to the number of dialogue turns in ascending order to obtain the first subset of dialogue data corresponding to the subset of dialogue data.

[0225] For example, the subset of dialogue data includes dialogue data with 1 round, dialogue data with 2 rounds, and dialogue data with 3 rounds. The dialogue data is arranged in ascending order of the number of rounds, with 1-round dialogue data first, followed by 2-round dialogue data, and finally 3-round dialogue data. If there are multiple dialogue data with the same number of rounds, for example, 3 dialogue data with 1 round, the internal order of these 3 dialogue data is not limited in this step.

[0226] For example, Figure 6 A schematic diagram illustrating sorting based on the first data dimension is provided. The default sorting of dialogue data in subset 1 is as follows: Figure 6 As shown in (a), dialogue data 1, dialogue data 2, and dialogue data 4 each have 1 round of dialogue. Dialogue data 3 and dialogue data 5 each have 2 rounds of dialogue. Dialogue data 6 each has 3 rounds of dialogue.

[0227] The electronic device sorts the six dialogue data in dialogue dataset 1 from the first data dimension, that is, according to the order of dialogue turns from smallest to largest. Dialogue data 4, which has one dialogue turn, follows dialogue data 3, which has two dialogue turns. The order of dialogue data 4 is then adjusted to precede that of dialogue data 3. The sorted first subset of dialogue data 1 can be found in [reference needed]. Figure 6 (b)

[0228] It is understandable that the above Figure 6 (a) provides an example where the sorting of one dialogue data in the dialogue data subset 1 needs to be adjusted. Even when multiple dialogue data in the dialogue data subset are sorted haphazardly according to the number of dialogue rounds, the dialogue data can still be sorted and arranged in ascending order of the number of dialogue rounds, based on the default sorting.

[0229] In this embodiment, a higher number of dialogue turns indicates more complex dialogue data. Therefore, sorting the dialogue data from easiest to most complex based on the number of dialogue turns achieves the effect of sorting the dialogue data from easiest to most complex. Training a large language model based on the sorted dialogue data allows the large language model to be trained using the idea of ​​learning from easy to difficult in a curriculum, which can improve the training efficiency of the large language model.

[0230] S303. The electronic device performs quantitative sorting on the dialogue data in the first dialogue data subset according to the second data dimension, and obtains the sorted second dialogue data subset.

[0231] For example, the second data dimension can be the number of non-empty slots contained in the last round of dialogue in the dialogue data (hereinafter referred to as the number of non-empty slots). The electronic device can sort the dialogue data in ascending order of the number of non-empty slots contained in the last round of dialogue in the dialogue data to obtain the second dialogue data subset corresponding to the dialogue data subset.

[0232] For example, the dialog data subset includes dialog data with 3 non-empty slots, dialog data with 2 non-empty slots, and dialog data with 4 non-empty slots. The dialog data with 2 non-empty slots is arranged first, followed by those with 3 non-empty slots, and then those with 4 non-empty slots are arranged last, in ascending order of the number of non-empty slots. If there are multiple dialog data sets with the same number of dialogue rounds, for example, if there are 3 dialog data sets with 3 non-empty slots, the internal order of these 3 dialog data sets is not limited in this step.

[0233] In some feasible embodiments, the electronic device may choose to sort only the first data dimension (that is, execute S302 after executing S301, but not S303), or sort only the second data dimension (execute S303 after executing S301, but not S302), or the electronic device may sort in combination with the first data dimension and the second data dimension (execute S302 and S303 after executing S301).

[0234] Figure 5 The document provides an example of how electronic devices can be sorted using a first data dimension and a second data dimension. Specifically, refer to [reference needed]. Figure 7 A schematic diagram illustrating sorting based on the second data dimension is provided. After sorting the dialogue data of subset 1 according to the first data dimension (number of dialogue turns) from smallest to largest, the following is obtained: Figure 7 The first subset of dialogue data shown in (a) is 1.

[0235] The dialogue data in the first subset 1 of the dialogue data includes: dialogue data 1 with 2 non-empty slots when the number of dialogue rounds is 1, dialogue data 2 with 2 non-empty slots, and dialogue data 4 with 4 non-empty slots; dialogue data 3 with 2 non-empty slots and dialogue data 5 with 1 non-empty slot when the number of dialogue rounds is 2; and dialogue data 6 with 3 dialogue rounds and 2 non-empty slots.

[0236] Assuming the dialogue data is arranged in ascending order of dialogue rounds, the order of non-empty slots is adjusted from smallest to largest. For dialogue data with 1 dialogue round, the data is already sorted by the number of non-empty slots in ascending order, requiring no adjustment. For dialogue data 5 with 2 dialogue rounds, the number of non-empty slots (1) is less than the number of non-empty slots (2) in dialogue data 3. Dialogue data 5 follows dialogue data 3. Therefore, the order of dialogue data 5 is adjusted to precede that of dialogue data 3. The sorted second subset of dialogue data 1 can be referenced. Figure 7 (b)

[0237] It is understandable that the above Figure 7 (b) provides an example where the sorting of a dialogue data in the dialogue data subset 1 needs to be adjusted. In the case where there are dialogue data with the same number of dialogue rounds in the dialogue data subset, and the sorting of the number of non-empty slots is disordered, the sorting of each dialogue data can still be sorted according to the order of the number of non-empty slots from smallest to largest, based on the sorting of the number of dialogue rounds.

[0238] In this embodiment, when the dialogue rounds include multiple rounds, the dialogues in the preceding rounds are subordinate dialogues and are in a completed state. Furthermore, during the training of the large language model, intent recognition (intent distribution and slot extraction) is only performed on the last round of dialogue; therefore, the focus can be on the number of non-empty slots in the last round. A higher number of non-empty slots indicates a greater range of needs expressed by the input content of the dialogue data, and thus a more complex dialogue data. Therefore, sorting the dialogue data from easiest to most difficult based on the number of non-empty slots achieves the effect of sorting the dialogue data from easiest to most difficult. Training the large language model based on the sorted dialogue data allows the large language model to be trained using a learning-from-easy-to-difficult approach, improving the training efficiency of the large language model.

[0239] S304. The electronic device acquires the quantized value of the dialogue data in the second dialogue data subset.

[0240] In this step, the quantization value refers to the quantification value of the learning difficulty / prediction difficulty (with the same complexity) of the large language model for dialogue data during the training process. Theoretically, the simpler the dialogue data, the lower the learning / prediction difficulty of the large language model, and the smaller the corresponding quantization value; the more complex the dialogue data, the higher the learning / prediction difficulty of the large language model, and the larger the corresponding quantization value.

[0241] In some feasible approaches, the number of dialogue turns and the number of non-empty slots in the last dialogue turn can, to some extent, characterize the complexity of the dialogue data. Therefore, in one example, the electronic device can quantize the first data dimension (number of dialogue turns) and the second data dimension (number of non-empty slots in the last dialogue turn) to obtain a first quantized value corresponding to each number of dialogue turns, and a second quantized value corresponding to different numbers of non-empty slots. For a given dialogue data set, based on its corresponding first and second quantized values, the quantized value corresponding to the dialogue data is obtained by calculating the average / cumulative sum / median / weighted sum / weighted average / weighted normalization, etc.

[0242] Alternatively, in some other feasible approaches, each dialogue data point in the second dialogue data subset can be sequentially input into a large language model for intent recognition, obtaining the intent recognition results (predicted results) for each dialogue data point. Based on the predicted results (predicted intent and predicted slot) and the actual results (actual intent and slot in the dataset) of each dialogue data point, the similarity between the predicted results and the actual results is calculated, and the quantization value of each dialogue data point is determined based on the similarity.

[0243] The higher the similarity of the dialogue data, the stronger the predictive ability of the large language model for that dialogue data, and the lower the quantitative value of the learning difficulty of that dialogue data; conversely, the lower the similarity of the dialogue data, the weaker the predictive ability of the large language model for that dialogue data, and the higher the quantitative value of the learning difficulty of that dialogue data.

[0244] Specifically, in combination Figure 5 The process of calculating the quantized value is given to illustrate this.

[0245] In this embodiment, the second dialogue data subset can be divided into quantization group 1 and quantization group 2. The purpose of grouping the second dialogue data subset is to obtain a training set and a test set.

[0246] On the one hand, quantization group 1 is used as the training set to train model 1 (a large language model). When model 1 reaches convergence, quantization group 2 is used as the test set to input into model 1 to obtain the quantization values ​​of each dialogue data in quantization group 2.

[0247] Convergence of Model 1 can be determined by several factors: First, the accuracy of Model 1's predictions compared to the labels in the training set can be calculated. If the accuracy does not improve, Model 1 is considered convergent. Second, the loss value of Model 1 can be calculated. If the loss value is less than a preset loss value, Model 1 is considered convergent. Third, Model 1 can be considered convergent if it has been trained a preset number of times.

[0248] The process of obtaining the quantization values ​​of each dialogue data in quantization group 2 includes:

[0249] Each dialogue data point in quantization group 2 is sequentially input into model 1 for intent recognition, obtaining the intent recognition result (prediction result) for each dialogue data point in quantization group 2. Based on the prediction results (predicted intent and predicted slot) and the actual results (actual intent and slot in the dataset) of each dialogue data point, the similarity between the prediction results and the actual results is calculated, and the quantization value of each dialogue data point is determined based on the similarity.

[0250] In one specific implementation, based on the inverse correspondence between similarity and quantization value, a calculation formula can be set between the quantization value b and the similarity a. For example, b = 1 - a. When the similarity is 0.8, the corresponding quantization value can be 1 - 0.8 = 0.2. When the similarity is 0.3, the corresponding quantization value can be 1 - 0.3 = 0.7.

[0251] Alternatively, based on the inverse correspondence between similarity and quantization value, a formula can be set to calculate the quantization value b and the similarity a. For example, b = 1 / a. When the similarity is 0.8, the corresponding quantization value can be 1 / 0.8 = 1.25. When the similarity is 0.3, the corresponding quantization value can be 1 / 0.3 = 3.33.

[0252] Alternatively, other calculation formulas based on the inverse correspondence between similarity and quantization value can be set. This embodiment does not limit the specific calculation method.

[0253] The higher the similarity of the dialogue data, the lower the quantification value of the dialogue data; the lower the similarity of the dialogue data, the higher the quantification value of the dialogue data.

[0254] On the other hand, quantization group 2 is used as the training set to train model 2 (a large language model). When model 2 reaches convergence, quantization group 1 is used as the test set to input into model 2 to obtain the quantization values ​​of each dialogue data in quantization group 1.

[0255] The process of obtaining the quantization values ​​of each dialogue data in quantization group 1 is similar to the process of obtaining the quantization values ​​of each dialogue data in quantization group 2, and will not be repeated here.

[0256] In this way, we can obtain the quantized values ​​of all dialogue data in the second dialogue data subset.

[0257] S305. The electronic device sorts the dialogue data in the second dialogue data subset according to the quantization value of the dialogue data from low to high, and obtains the sorted third dialogue data subset.

[0258] For example, refer to Figure 8 A schematic diagram of sorting based on quantized values ​​is provided. After sorting the dialogue data of subset 1 based on the second data dimension (number of non-empty slots) from smallest to largest, the following is obtained: Figure 8 The second dialog data subset 1 is shown in (a).

[0259] The dialogue data in the second subset 1 of the dialogue data includes: dialogue data 1 with a quantization value of 0.3, dialogue data 2 with a quantization value of 0.2, dialogue data 4 with a quantization value of 0.4, dialogue data 5 with a quantization value of 0.3, dialogue data 3 with a quantization value of 0.5, and dialogue data 6 with a quantization value of 0.6.

[0260] Within the second subset of dialogue data (1), the quantization values ​​are sorted from lowest to highest. In dialogue data with one turn, dialogue data 2 with a quantization value of 0.2 follows dialogue data 1 with a quantization value of 0.3; therefore, dialogue data 2 is moved to precede dialogue data 1. For dialogue data with the same number of turns / number of non-empty slots, other dialogue data are sorted from lowest to highest quantization value without adjustment. The sorted third subset of dialogue data (1) can be found here. Figure 8 (b)

[0261] It is understandable that the above Figure 8 (b) provides an example where the sorting of a dialogue data in the dialogue data subset 1 needs to be adjusted. When there is a disordered sorting of quantization values ​​on the basis that the dialogue data subset has the same number of dialogue rounds / the same number of non-empty slots, the sorting of each dialogue data can still be sorted according to the order of quantization values ​​from low to high on the basis of the sorting of dialogue rounds and the number of non-empty slots.

[0262] In this embodiment, a lower quantization value of the dialogue data indicates lower learning difficulty and a stronger learning / prediction ability of the large language model for that data; conversely, a higher quantization value indicates higher learning difficulty and a weaker learning / prediction ability of the large language model. Sorting the dialogue data from low to high quantization values ​​achieves the effect of ranking the dialogue data from easy to difficult. Training the large language model based on this ranked dialogue data allows for a learning-from-easy-to-difficult approach, improving training efficiency.

[0263] After sorting and organizing the data subsets corresponding to each data type in the above steps, we obtain the third dialogue data subset corresponding to each data type. Data types include, for example... Figure 5The diagram illustrates single-turn, follow-up, domain-based, secondary execution, intent inheritance, and slot inheritance. Single-turn dialogue data is sorted to obtain the corresponding third dialogue data subset 1; follow-up dialogue data is sorted to obtain the corresponding third dialogue data subset 2; domain-based dialogue data is sorted to obtain the corresponding third dialogue data subset 3; secondary execution dialogue data is sorted to obtain the corresponding third dialogue data subset 4; intent inheritance dialogue data is sorted to obtain the corresponding third dialogue data subset 5; and slot inheritance dialogue data is sorted to obtain the corresponding third dialogue data subset 6. Each data type's corresponding third dialogue data subset includes multiple dialogue data sets.

[0264] In the field of large language model technology, batch processing is commonly used to process training datasets. Specifically, multiple batches can be pre-defined, and the size of each batch can be set. For example, the size of each batch can be a power of 2 (e.g., 16, 32, 64, 128, 256, 512). During large language model training, data from each batch is sequentially acquired for training, which optimizes the model's training efficiency and performance.

[0265] After obtaining the third subset of dialogue data corresponding to each data type, the electronic device executes:

[0266] S306: Based on the third dialogue data subset corresponding to each data type, the batch is divided into multiple batches with filled data for model training.

[0267] In this embodiment, the size of each batch can be set, for example, each batch size is 32. The number of batches can also be set, such as... Figure 5 As shown, in this embodiment, the number of batches can be set to 6. Optionally, the size of each batch can also be different.

[0268] Depending on the batch size, a certain amount of dialogue data can be divided into each batch from the third dialogue data subset of each data type, according to the order of the dialogue data.

[0269] For example, batch1 (b1) can store 30 dialogue data entries. With 6 data types, 5 dialogue data entries (originally ordered 1-5) can be sequentially retrieved from the third subset of each data type's dialogue data and placed into b1, forming data block b1. Similarly, batch2 (b2) can also store 30 dialogue data entries. With 6 data types, 5 dialogue data entries (originally ordered 6-10) can be sequentially retrieved from the third subset of each data type's dialogue data and placed into b2, forming data block b2.

[0270] In some embodiments, as the dialogue data is sorted, the later dialogue data becomes increasingly complex, and the number of dialogue data that can be stored in a batch may decrease. For example, a batch size of 6 can store 18 dialogue data entries. If there are 6 data types, then from the third subset of dialogue data of each data type, in order of dialogue data, based on the dialogue data corresponding to b5, 3 dialogue data entries can be sequentially obtained and placed into b6 to form a data block b6.

[0271] Based on the size of each batch, and following the principle of balanced sampling from multiple data types, a certain amount of dialogue data is sequentially obtained from the third subset of dialogue data for each data type, according to the order of the dialogue data, until all batches are filled. Each batch is then sequentially input into the large language model for model training.

[0272] It is understandable that S306 can be executed before model training; or, S306 can be executed in advance to obtain and store multiple batch blocks with padded data, and when model training is required, multiple batch blocks with padded data can be obtained for model training.

[0273] In this implementation, due to the varying difficulty levels among the dialogue data, directly inputting the dialogue data into the large language model for training without differentiating between easy and difficult data may lead to the large language model over-training on simple data and under-training on complex data, resulting in poor training performance. In the above embodiment, by sorting and organizing the training dataset according to the first data dimension, the second data dimension, and the quantified value of (learning difficulty), the learning priority of each dialogue data in the training dataset is defined. Learning simpler dialogue data first, followed by more complex dialogue data, allows the large language model to be trained using a learning-from-easy-to-difficulty approach, thus improving training efficiency.

[0274] above Figure 4 and Figure 5The embodiments describe the acquisition of the training dataset and the further sorting and organization of the training dataset in the model training method. After obtaining multiple batches of padded data, the large language model is trained based on these batches, including:

[0275] The electronic device sequentially inputs the batch blocks, after filling them with data, into the initial large language model for model training, in the order of the batch blocks, to obtain the trained large language model.

[0276] In this embodiment, during training, the large language model can sequentially read each dialogue data in the batch processing block according to the order of the batch processing block, perform intent recognition on the dialogue data, and obtain the intent recognition result (prediction result) corresponding to the dialogue data.

[0277] The training mode of the large language model can be a traditional training mode such as supervised learning or semi-supervised learning. In this embodiment, the specific training mode of the large language model is not limited.

[0278] In this embodiment, model training can be terminated when the preset number of training iterations is reached, resulting in a trained large language model. Alternatively, a validation dataset can be input into the large language model. Based on the large language model's prediction results for the validation dataset and the actual results of the validation dataset, the model's accuracy and recall can be calculated. If the accuracy and recall no longer improve, the trained large language model is obtained. Alternatively, the loss value of the large language model can be calculated. If the loss value is less than a preset loss value or no longer decreases, the trained large language model is obtained.

[0279] In this embodiment, the dialogue data in the batch processing block, as well as the dialogue data involved in the data processing in the above embodiments, specifically refers to single-intent data. Training a large language model based on single-intent data can effectively handle single-intent intent recognition. However, in real-world human-computer dialogue scenarios, the user's input (dialogue data) may include multiple intents. To make the large language model applicable to recognition scenarios with input containing multiple intents, in some embodiments... Figure 9 A flowchart illustrating another model training method is provided, including:

[0280] S401. The electronic device sequentially inputs the batch processing blocks after filling the data into the initial large language model for model training according to the order of the batch processing blocks, and obtains the candidate large language model.

[0281] Among them, the candidate large language model refers to the model trained based on dialogue data with a single intent.

[0282] In some embodiments, the electronic device may also input the training dataset obtained in S203 into the initial large language model for model training to obtain a candidate large language model. In this embodiment, the training dataset obtained in S203 and the dialogue data in the batch block obtained in S306 both refer to single-intent dialogue data.

[0283] Referring to the above embodiments, model training can be terminated when the preset number of training iterations is reached, yielding a candidate large language model. Alternatively, a validation dataset can be input into the large language model. Based on the large language model's prediction results on the validation dataset and the actual results on the validation dataset, the model's accuracy and recall are calculated. A candidate large language model is obtained when the accuracy and recall no longer improve. For example, the model's accuracy threshold can be set to 90%, meaning that model training terminates when the model's accuracy reaches 90%, yielding a candidate large language model. Alternatively, the loss value of the large language model can be calculated. A candidate large language model is obtained when the loss value is less than a preset loss value or when the loss value no longer decreases.

[0284] The S401 stage, which trains the model based on single-intent dialogue data, can be considered the first training stage. After obtaining a trained single-intent model, multi-intent recognition training is performed. This multi-intent recognition training can be considered the second training stage. To prevent the model from forgetting single-intent knowledge during the second training stage, single-intent dialogue data is added to the second training set. For example, the same number of data points as the multi-intent data can be randomly sampled from the single-intent dialogue dataset and then mixed with the multi-intent data to obtain the training set for the second stage training of the large language model.

[0285] S402. The electronic device randomly samples a preset number of dialogue data from the training dataset.

[0286] In this embodiment, the training dataset can be the training dataset obtained in S203, or it can be the collection of the third dialogue data subsets corresponding to each data type obtained in S305.

[0287] The electronic device randomly samples a predetermined number of dialogue data from the training dataset as single-intent dialogue data for the second training phase.

[0288] S403, The electronic device acquires a preset number of dialogue data with dual intents.

[0289] Electronic devices can acquire dialogue data with a preset number of two intentions from open-source datasets; or, electronic devices can generate dialogue data with a preset number of two intentions based on a preset model.

[0290] For example, the dialogue data for a dual intent could be: {"dialog":[{"user":"Query the weather in City A today and the flights from City B to City A today","state":"Query the weather (location = City A, date = today (20240925))","state":"Query the flights (departure point = City B, destination = City A, date = today (20240925))","assistant":"The weather in City A today is sunny, with a high of 28°C and a low of 21°C. Flights from City A to City B today include: [flight card]"}, where the flight card can include flight information from City A to City B after the query time, such as September 25, 2024.

[0291] The input content of the dialogue data is "Query the weather in City A today and the flights from City B to City A today"; after intent recognition based on the input content, the intents obtained include "Query the weather" and "Query flights". The slots include "Location = City A, Date = Today (20240925), Departure City = City B, Destination = City A"; the corresponding response content is "The weather in City A today is sunny, with a high of 28℃ and a low of 21℃. Today's flights from City A to City B include: [Flight Card]".

[0292] S404. The electronic device uses single-intent dialogue data and dual-intent dialogue data as inputs to the candidate large language model for model training, and obtains the target large language model.

[0293] In some embodiments, the electronic device inputs single-intent dialogue data and dual-intent dialogue data into a candidate large language model for model training to obtain a target large language model.

[0294] Referring to the above embodiments, model training can be terminated when the preset number of training iterations is reached, yielding the target large language model. Alternatively, a validation dataset can be input into the large language model. Based on the large language model's prediction results for the validation dataset and the actual results of the validation dataset, the model's accuracy and recall can be calculated. The target large language model is obtained when accuracy and recall no longer improve. Since the complexity of dialogue data with two intentions is higher than that with single intentions, the accuracy threshold for a large language model suitable for two-intention dialogue scenarios can be lower than that for a large language model suitable for single-intention dialogue scenarios. For example, the accuracy threshold can be set to 85%. That is, model training terminates when the model's accuracy reaches 85%, yielding the target large language model. Alternatively, the loss value of the large language model can be calculated. The target large language model is obtained when the loss value is less than a preset loss value or when the loss value no longer decreases.

[0295] The S401 stage, which trains the model based on single-intent dialogue data, can be considered the first training stage. After obtaining a trained single-intent model, multi-intent recognition training is performed. This multi-intent recognition training can be considered the second training stage. To prevent the model from forgetting single-intent knowledge during the second training stage, single-intent dialogue data is added to the second training set. For example, the same number of data points as the multi-intent data can be randomly sampled from the single-intent dialogue dataset and then mixed with the multi-intent data to obtain the training set for the second stage training of the large language model.

[0296] In this embodiment, a two-stage fine-tuning method is introduced. In the first training stage, the large language model is fine-tuned using only single-intent dialogue data. In the second training stage, the large language model is trained using a mixture of single and dual-intent dialogue data. The trained target large language model performs better on the same test set, especially for intent recognition of dual-intent dialogue data, exhibiting higher accuracy and recall. Simulation experiments show that using the target large language model improves the metrics (accuracy and recall) for all intents and slots on the gold-standard test set by 1%-3%, with some slots showing an 8% improvement in metrics (accuracy and recall).

[0297] Combining the above processing of the original dataset with the specific model training process, Figure 10 A complete schematic diagram of the model training method is provided, including:

[0298] S501, The electronic device acquires the first dialogue data.

[0299] Here, the first dialogue data refers to dialogue data with a single intent. The first dialogue data can be single-intent dialogue data from the original training dataset.

[0300] For details, please refer to the embodiments provided in S201 above, which will not be repeated here.

[0301] S502, The electronic device performs data correction processing on the first dialogue data to obtain the first dialogue data after data correction processing.

[0302] The data correction process includes slot replenishment, correction filtering, splitting data across multiple rounds, and diversity verification. For details, please refer to the data correction process described in the embodiment provided in S201 above; it will not be repeated here.

[0303] S503. The electronic device divides the first dialogue data after data correction processing into groups based on at least one preset data dimension, resulting in multiple groups.

[0304] The preset data dimensions include data type, intent category, number of dialogue turns, and non-empty slot combinations. For details, please refer to the data grouping method in the embodiment provided in S202 above; it will not be elaborated upon here.

[0305] S504. The electronic device samples the n groups corresponding to the original dataset and merges the sampled dialogue data to obtain the training dataset.

[0306] For details, please refer to the group sampling method in the embodiment provided in S203 above, which will not be repeated here.

[0307] S505. The electronic device sorts and organizes the dialogue data in the training dataset to obtain the first training dataset used in the first training phase.

[0308] The sorting and ranking can include sorting based on the number of dialogue turns in the dialogue data from smallest to largest, sorting based on the number of non-empty slots in the last dialogue turn from smallest to largest, and sorting based on the quantization value of the representation learning difficulty from low to high. For details, please refer to the sorting and ranking methods in the embodiments provided in S301-S305 above, which will not be elaborated here.

[0309] S506. The electronic device uses the first training dataset as input to the large language model for the first stage of training of the model, and obtains the candidate large language model.

[0310] The electronic device can divide the first training dataset into multiple batch processing blocks. Following the order of the batch processing blocks, the data-filled batch processing blocks are sequentially input into the initial large language model for model training, resulting in candidate large language models. For details, please refer to the methods provided in the embodiments of S306 and S401 above; they will not be elaborated upon here.

[0311] S507. The electronic device randomly samples a preset number of first dialogue data from the training dataset.

[0312] The first dialogue data is single-intent dialogue data. For details, please refer to the method in the embodiment provided in S402 above; it will not be elaborated upon here.

[0313] S508, The electronic device acquires a preset number of second dialogue data.

[0314] The second dialogue data is dialogue data with dual intents. For details, please refer to the method in the embodiment provided in S403 above; it will not be elaborated upon here.

[0315] S509: The electronic device uses the first dialogue data and the second dialogue data as input to the large language model for the second stage of training of the model, and obtains the target large language model.

[0316] For details, please refer to the methods in the embodiments provided in S404 above, which will not be repeated here.

[0317] In this embodiment, the electronic device analyzes data dimensions, extracting four dimensions: intent classification, data type, number of dialogue turns, and non-empty slot combinations. Then, through grouping and within-group sampling, it ensures that the dialogue data in the constructed training dataset has a similar number of data points across multiple dimensions, maintaining a balanced distribution. Furthermore, it adopts a learning-from-easy-to-difficult approach, sorting data within the same data type and allowing the model to learn from easier to more difficult dialogue data. This order of training dialogue data improves the efficiency of the large language model's training, thereby increasing its precision and recall. This embodiment also incorporates a two-stage fine-tuning method. In the first training stage, only single-intent dialogue data is used for fine-tuning the large language model. In the second training stage, a mixture of single and dual-intent dialogue data is used. The trained target large language model performs better on the same test set, especially for intent recognition of dual-intent dialogue data, exhibiting higher accuracy and recall.

[0318] Figure 11 A schematic diagram of another possible structure of the electronic device involved in the above embodiments is shown. Figure 11 The electronic device 1100 shown includes a processing module 1101, a communication module 1102, and a storage module 1103.

[0319] The processor 1101 may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0320] For example, the processing module 1101 can be as follows: Figure 3 The processor 101 shown; the communication module 1102 can be as follows: Figure 3 The communication interface 103 shown; the storage module 1103 can be as follows: Figure 3 The memory 102 shown. The electronic device provided in this application embodiment can be Figure 3 The electronic device 100 shown.

[0321] This application also provides a chip system (e.g., a system-on-a-chip (SoC)). Figure 12 As shown, the chip system includes at least one processor 1201 and at least one interface circuit 1202. The processor 1201 and the interface circuit 1202 are interconnected via lines. For example, the interface circuit 1202 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1202 can be used to send signals to other devices (e.g., the processor 1201 or the camera of an electronic device). Exemplarily, the interface circuit 1202 can read instructions stored in the memory and send those instructions to the processor 1201. When the instructions are executed by the processor 1201, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application embodiment.

[0322] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device 100 in the above method embodiment.

[0323] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the electronic device 100 in the above method embodiments. For example, the computer may be the aforementioned electronic device 100.

[0324] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0325] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0326] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0327] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0328] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0329] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a large language model, characterized in that, The large language model is applied to human-computer dialogue scenarios, and the method includes: An electronic device acquires a raw dataset; the raw dataset includes multiple dialogue data; the dialogue data is used to characterize the data contained in the dialogue in a human-computer dialogue scenario; The electronic device categorizes the dialogue data in the original dataset into multiple groups based on preset data dimensions. Each group corresponds to one data dimension or a combination of one or more data dimensions. The preset data dimensions include data type, intent classification, number of dialogue turns, and non-empty slot combination, each with a corresponding priority. The multiple groups are obtained by categorizing the dialogue data in the original dataset according to the priority of each data dimension from high to low. The priority of data type is higher than that of intent classification, which is higher than that of number of dialogue turns, which is higher than that of non-empty slot combination. The electronic device samples data from the multiple groups respectively, and uses the collection of sampled dialogue data as a training dataset. The dialogue data in the training dataset is sorted from low to high according to the complexity of the dialogue data. The complexity of the dialogue data is used to characterize the accuracy of the large language model in predicting the intent of the dialogue data. The smaller the number of dialogue rounds and the smaller the number of non-empty slots in the last round of dialogue, the simpler the dialogue data, the higher the accuracy of intent prediction, and the lower the complexity of the dialogue data. The electronic device inputs the first dialogue data from the training dataset into the large language model for the first stage of training. When the model reaches convergence, a candidate large language model is obtained. The first dialogue data is single-intent dialogue data. The electronic device randomly samples a preset number of first dialogue data from the training dataset; the electronic device acquires the preset number of second dialogue data; the second dialogue data is dialogue data with two intents; The electronic device inputs the first dialogue data and the second dialogue data into the candidate large language model for the second stage of training. When the model reaches convergence, the trained large language model is obtained.

2. The method according to claim 1, characterized in that, The electronic device categorizes the dialogue data in the original dataset into groups based on a preset data dimension, resulting in multiple groups including: The electronic device divides the dialogue data in the original dataset into groups according to the data type to obtain a first group; the data type includes at least two of the following: single turn, follow-up question, domain cutting, secondary execution, intent inheritance, and slot inheritance. Under the same data type, the electronic device divides the dialogue data in each of the first groups into second groups according to the intent classification; Under the same intent classification, the electronic device divides the dialogue data in each of the second groups according to the number of dialogue rounds to obtain a third group; Under the same number of dialogue rounds, the electronic device divides the dialogue data in each of the third groups according to the non-empty slot combination to obtain the multiple groups.

3. The method according to claim 1 or 2, characterized in that, The preset data dimension includes data types, which include a first data type and a second data type. The electronic device samples data from the multiple groups respectively, and uses the combined set of sampled dialogue data as a training dataset, including: The electronic device samples a first number of dialogue data from the group corresponding to the first data type; The electronic device samples a second number of dialogue data from the groups corresponding to the second data type; The electronic device uses the collection of sampled dialogue data as the training dataset. Wherein, the first quantity is greater than the second quantity, the level of detail of the dialogue data represented by the first data type is lower than the level of detail of the dialogue data represented by the second data type, the first data type includes single turn, follow-up question or domain cut, and the second data type includes secondary execution, intent inheritance or slot inheritance.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The electronic device performs data correction processing on the training dataset to obtain a data-corrected training dataset. The electronic device divides the dialogue data in the original dataset into groups based on a preset data dimension, resulting in multiple groups, including: The electronic device divides the dialogue data in the training dataset after data correction processing into multiple groups based on the preset data dimension.

5. The method according to claim 4, characterized in that, The data correction process includes at least one of slot replenishment, correction filtering, splitting multiple rounds of data, and diversity verification. The slot replenishment includes replenishing the slots of dialogue data that are empty. The correction filtering includes correcting the slots of dialogue data with abnormal slots, correcting the intent of dialogue data with abnormal intent recognition, or removing abnormal dialogue data. The splitting of multi-turn data includes splitting dialogue data containing multiple turns of dialogue into dialogue data of a single turn of dialogue or dialogue data of a two-turn dialogue; The diversity verification includes removing dialogue data whose sentence structure overlap is higher than a preset overlap value.

6. The method according to claim 1 or 2, characterized in that, The method further includes: The electronic device sorts the dialogue data under each data type according to the complexity of the dialogue data from low to high, to obtain the training dataset.

7. The method according to claim 6, characterized in that, The electronic device sorts the dialogue data under each data type according to the complexity of the dialogue data from low to high, to obtain the training dataset, which includes: The electronic device divides the dialogue data into different data types to obtain a subset of dialogue data corresponding to each data type. The electronic device sorts the dialogue data in each subset of dialogue data according to the complexity of the dialogue data from low to high, and uses the combined set of all sorted dialogue data subsets as the training dataset.

8. The method according to claim 7, characterized in that, The electronic device sorts the dialogue data in each subset of dialogue data according to the complexity of the dialogue data from low to high, and uses the union of all sorted dialogue data subsets as the training dataset, including: For each data type of dialogue data subset, the electronic device sorts the dialogue data in the dialogue data subset according to the order of the number of dialogue turns in ascending order, and obtains the sorted first dialogue data subset. The electronic device sorts the dialogue data in the first dialogue data subset based on the number of non-empty slots in the last round of dialogue in the dialogue data from smallest to largest, and obtains the sorted second dialogue data subset. The collection of the second dialogue data subsets corresponding to all data types is used as the training dataset.

9. The method according to claim 7 or 8, characterized in that, The electronic device sorts the dialogue data in each subset of dialogue data according to the complexity of the dialogue data from low to high, and uses the union of all sorted dialogue data subsets as the training dataset, including: The electronic device acquires the quantized value of the dialogue data in the dataset to be sorted; the quantized value refers to the quantized value of the prediction accuracy of the large language model for the dialogue data during the training process of the large language model. The electronic device sorts the dialogue data in the dataset to be sorted according to the quantization value of each dialogue data from low to high, and obtains the sorted third subset of dialogue data. The collection of third-party dialogue data subsets corresponding to all data types is used as the training dataset; The dataset to be sorted includes each of the aforementioned subsets of dialogue data, or each of the second subsets of dialogue data.

10. The method according to claim 9, characterized in that, The electronic device acquires quantized values ​​of dialogue data in the second dialogue data subset, including: The electronic device divides the second subset of dialogue data into a first quantization group and a second quantization group; Using the first quantized group as training data and the second quantized group as test data, the intent prediction results of each dialogue data in the second quantized group are obtained through the first preset model. Based on the similarity between the intent prediction results of each dialogue data in the second quantization group and the actual intent results of each dialogue data in the second quantization group, the quantization value of each dialogue data in the second quantization group is obtained. Using the second quantized group as training data and the first quantized group as test data, the intent prediction results of each dialogue data in the first quantized group are obtained through the second preset model. Based on the similarity between the intent prediction results of each dialogue data in the first quantization group and the actual intent results of each dialogue data in the first quantization group, the quantization value of each dialogue data in the first quantization group is obtained.

11. The method according to claim 7 or 8, characterized in that, The electronic device sequentially inputs dialogue data from the training dataset into the large language model for the first stage of training, according to the order of dialogue data complexity from high to low. When the model reaches convergence, a candidate large language model is obtained, including: The electronic device divides the training dataset into data types to obtain dialogue datasets corresponding to each data type; the dialogue data in the dialogue datasets are ordered from low to high complexity. The electronic device divides the batch processing blocks sequentially based on the dialogue dataset corresponding to each data type to obtain multiple batch processing blocks filled with data. The electronic device sequentially uses each batch processing block as input to the large language model for the first stage of training, and obtains the candidate large language model when the model reaches convergence.

12. A method for acquiring training data, characterized in that, The training data is used to train a large language model, which is then applied to human-computer dialogue scenarios. The method includes: An electronic device acquires a raw dataset; the raw dataset includes multiple dialogue data; the dialogue data is used to characterize the data contained in the dialogue in a human-computer dialogue scenario; The electronic device divides the dialogue data in the original dataset into multiple groups based on preset data dimensions. Each group corresponds to one data dimension or a combination of data dimensions. The preset data dimensions include data type, intent category, number of dialogue turns, and non-empty slot combination. When the preset data dimensions include multiple items, each data dimension has a corresponding priority. The multiple groups are obtained by dividing the dialogue data in the original dataset into groups according to the priority of each data dimension from high to low. The priority of data type is higher than the priority of intent category, the priority of intent category is higher than the priority of number of dialogue turns, and the priority of number of dialogue turns is higher than the priority of non-empty slot combination. The electronic device samples data from the multiple groups respectively, and uses the collection of sampled dialogue data as the training dataset of the large language model; the dialogue data in the training dataset is sorted from low to high according to the complexity of the dialogue data. The complexity of the dialogue data is used to characterize the accuracy of the large language model in predicting the intent of the dialogue data. The smaller the number of dialogue rounds and the smaller the number of non-empty slots in the last round of dialogue, the simpler the dialogue data, the higher the accuracy of intent prediction, and the lower the complexity of the dialogue data. The training dataset includes first dialogue data and second dialogue data, wherein the first dialogue data is single-intent dialogue data and the second dialogue data is dual-intent dialogue data; the training dataset is used to train a large language model, wherein the training process includes a first-stage training of the large language model based on the first dialogue data in the training dataset, and a second-stage training of the converged large language model based on the first dialogue data and the second dialogue data when the model reaches convergence.

13. An intent recognition method, characterized in that, The method includes: Electronic devices acquire user input statements in human-computer dialogue scenarios; The electronic device inputs the input statement into a preset large language model and obtains the intent recognition result corresponding to the input statement; the intent recognition result includes the intent and slot corresponding to the input statement. Based on the intent recognition result, the electronic device outputs the response content corresponding to the input statement; The training method for the preset large language model is provided by any one of claims 1-11.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program performs the steps of the method described in any one of claims 1-13.

16. A computer program product, comprising a computer program, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-13.

Images