A method, system, device and storage medium for constructing a large model multi-round dialogue

Abstract

A method, system, device and storage medium for constructing a large-model multi-round dialogue, relating to the field of artificial intelligence. In this method, a question-answer pair data set is obtained, and the question-answer pair data set is adjusted based on Gaussian distribution to obtain a total sample data set; a first preset number is determined according to a preset batch size, and the first preset number of question-answer pairs are randomly extracted from the total sample data set to form a sub-sample data set; the sub-sample data set is adjusted to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set; based on the target sub-sample data set, a first multi-round question-answer pair is formed. The technical solution provided in this application is implemented, the data set is adjusted based on Gaussian distribution, and the data source quality of the multi-round dialogue is optimized by random sampling and adjusting the sub-samples, which significantly improves the coherence of the multi-round dialogue and the response accuracy of the model.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence, and specifically to a method, system, device and storage medium for constructing a large-model multi-round dialogue. Background Art

[0002] The multi-round dialogue capability of large models is increasingly valued in the field of natural language processing. In order to achieve more natural, coherent and logical dialogue, the model needs to effectively learn from a large amount of dialogue data.

[0003] In the prior art, there are two main methods for constructing large models to fine-tune multi-round dialogues: one is the manual strategy combination construction method, which combines single question-answer pairs through preset fixed strategies to form a multi-round dialogue set; the other is the large model generation method, which uses the extraction and combination capabilities of the large model to generate multi-round dialogue samples from the source text. However, both methods have certain limitations. Specifically, the manual strategy combination construction method lacks sufficient generalization capabilities, and the generated multi-round dialogues perform poorly in terms of diversity. At the same time, due to the low efficiency of manual strategy combination and a large amount of redundancy in the combination process, this leads to a serious waste of training resources. As for the large model generation method, although it can use the powerful capabilities of the large model for dialogue generation, the process consumes huge computing resources and is costly. In addition, the multi-round dialogue samples generated by this method also have the problem of insufficient diversity, especially when switching topics, the model is often prone to divergence and inaccuracy.

[0004] Therefore, how to overcome these shortcomings of the existing technology and propose a more efficient, diverse and accurate method for constructing large-model multi-round dialogues has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] The present application provides a method, system, device and storage medium for constructing a large-model multi-round dialogue, which adopts a batch random mixing method to generate multi-round dialogues, which can better improve the diversity and richness of multi-round dialogue generation, and randomly select question-answer pairs through Gaussian distribution to ensure that the selected subset is the same distribution as the overall sample space or close to the overall sample space distribution, thereby reducing the diversity of samples and improving the accuracy of the fine-tuning model.

[0006] In a first aspect of the present application, a method for constructing a large-model multi-round dialogue is provided, which is applied to a large-model training platform. The construction method includes: obtaining a question-answer pair dataset, adjusting the question-answer pair dataset based on a Gaussian distribution to obtain a total sample dataset; determining a first preset number according to a preset batch size, and randomly extracting a first preset number of question-answer pairs from the total sample dataset to form a sub-sample dataset; adjusting the sub-sample dataset to obtain a target sub-sample dataset so that the target sub-sample dataset conforms to the spatial distribution of the total sample dataset; and forming a first multi-round question-answer pair based on the target sub-sample dataset.

[0007] By adopting the above technical solution, the question-answer pair dataset is adjusted based on the Gaussian distribution, so that the final total sample dataset can increase the diversity and coverage of the data while retaining the characteristics of the original data. This adjustment helps the large model learn a wider and more comprehensive dialogue mode and context, thereby improving the generalization ability of dialogue generation. The sub-sample dataset is adjusted to conform to the spatial distribution of the total sample dataset. This step ensures that the overall distribution characteristics of the data will not change significantly even when processing a smaller data subset. This is crucial to maintain the consistency and stability of model training and helps to avoid deviation or overfitting of the model during training. The first multi-round question-answer pair is formed based on the adjusted target sub-sample dataset. This step is directly related to the final multi-round dialogue generation quality. Since the target sub-sample dataset is optimized in terms of diversity and distribution consistency, the multi-round question-answer pairs generated based on these data will be more natural and smooth, and can better simulate the dialogue process in real scenarios. Through the above construction method, the large model can be exposed to more diverse dialogue scenarios and contexts during training, thereby enhancing its generalization ability. This means that the model can respond more flexibly and generate more accurate answers when faced with unknown or new dialogue tasks.

[0008] Optionally, adjusting the question and answer pair data set based on Gaussian distribution to obtain a total sample data set includes: dividing the question and answer pair data set into multiple first intervals, counting the first theoretical probability and the first actual sample number ratio of the first target interval, adjusting the number of question and answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and forming a total sample data set based on all adjusted question and answer pairs in the first target interval, the first target interval being any first interval among the multiple first intervals.

[0009] By adopting the above technical solution, the number of question-answer pairs in each interval (or category) can be made more consistent with the Gaussian distribution based on the adjustment of the Gaussian distribution. Adjusting the question-answer pair data set to a Gaussian distribution helps the model learn more common and general features, thereby improving the generalization ability of the model. During the model training process, whether the data distribution conforms to the Gaussian distribution has an important impact on the training efficiency and convergence speed. By adjusting the data distribution, the fluctuation and instability in the training process caused by uneven data distribution can be reduced, thereby speeding up the training speed and optimizing the training effect. Data distribution that does not conform to the Gaussian distribution may also cause the model to over-focus on certain categories of data during training and ignore other categories of data, thereby increasing the risk of overfitting. Through adjustments based on the Gaussian distribution, the model can consider the data of each category more comprehensively during the training process, reducing the occurrence of overfitting.

[0010] Optionally, the question-answer pair data set is divided into multiple first intervals, the first theoretical probability and the first actual sample number ratio of the first target interval are counted, the number of question-answer pairs in the first target interval is adjusted according to the first theoretical probability and the first actual sample number ratio, and the total sample data set is formed according to all adjusted question-answer pairs in the first target interval, including: counting the text lengths of the question-answer pairs in the question-answer pair data set to form a text length set, determining the first text interval range according to the first text length and the second text length, the first text length being the minimum text length in the text length set, and the second text length being the maximum text length in the text length set; and calculating the mean of the text lengths in the text length set. and variance, evenly divide the first text interval range into multiple first intervals, divide the question and answer pairs in the question and answer pair data set into corresponding first intervals according to the text length, and count the number of question and answer pairs in each first interval; obtain the first maximum value and the first minimum value of the text interval range of the first target interval, obtain the first theoretical probability according to the first maximum value, the first minimum value, the mean, the variance and the Gaussian distribution function, and obtain the first actual sample number ratio according to the number of question and answer pairs in the first target interval; adjust the number of question and answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and form a total sample data set according to all the adjusted question and answer pairs in the first target interval.

[0011] By adopting the above technical solution, the text length of the question-answer pair is counted and divided into multiple intervals, so that the distribution of data in the dimension of text length can be clearly seen. Subsequently, the number of question-answer pairs in each interval is adjusted according to the characteristics of Gaussian distribution (mean and variance), so that the overall distribution is more consistent with Gaussian distribution, reducing the interference of extreme length text on model training. Data distribution that does not conform to Gaussian distribution often causes fluctuations in the model during training, and even in some extreme cases, causes the model to fail to converge. By adjusting the data distribution so that the data in each interval is more consistent with Gaussian distribution, it helps the model maintain stability during training, improves convergence speed and training effect. The generalization ability of a model refers to the ability of a model to perform well on unseen data. By adjusting the data distribution, the model is exposed to more diverse data during training, which helps the model learn more common and general features, thereby improving its performance on new data. When the data distribution does not conform to Gaussian distribution, the model may over-focus on certain specific types of data and ignore other types of data, thereby increasing the risk of overfitting. By adjusting the data distribution, the model can consider various types of data more comprehensively during training, which helps reduce the occurrence of overfitting.

[0012] Optionally, adjusting the number of question and answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio includes: comparing the first theoretical probability with the first actual sample number ratio; when the first theoretical probability is greater than the first actual sample number ratio, calculating a second preset number, randomly selecting a second preset number of samples from the first target interval for copying and updating the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than a first threshold; when the first theoretical probability is less than the first actual sample number ratio, calculating a third preset number, randomly taking a third preset number of samples from the first target interval and updating the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than the first threshold.

[0013] By adopting the above technical solution, comparing the first theoretical probability (calculated based on Gaussian distribution) and the first actual sample number ratio, the intervals with too many or too few samples can be clearly identified. According to the comparison results, corresponding adjustments are made so that the number of samples in each interval is more consistent with the Gaussian distribution, reducing the impact of extreme cases on model training. The data distribution state of the data set has an important impact on the model training effect. When the distribution of the data set in each feature dimension conforms to the Gaussian distribution, the model can learn the characteristics of the data more comprehensively, thereby improving its generalization ability on new data. In addition, the data set that conforms to the Gaussian distribution also helps to reduce fluctuations and instability during model training and improve training efficiency. In the embodiment of the present application, when the theoretical probability is greater than the actual sample number ratio, the number is increased by copying the sample; when the theoretical probability is less than the actual sample number ratio, the number is reduced by removing the sample. This flexible adjustment strategy can be selected according to the actual situation to achieve the best data distribution effect. During the adjustment process, by calculating the second preset number or the third preset number and setting the first threshold to control the adjustment range, it can be ensured that the adjusted data distribution is neither too concentrated nor too dispersed, and remains within a reasonable range. This precise control helps to avoid the negative impact of excessive adjustment.

[0014] Optionally, adjusting the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set includes: counting the text lengths of the question-answer pairs in the sub-sample data set to form a sub-text length set, determining the second text interval range according to the third text length and the fourth text length, and adjusting the second text interval range according to the mean and the variance to obtain a third text interval range, the third text length being the minimum text length in the sub-text length set, and the fourth text length being the maximum text length in the sub-text length set; evenly dividing the third text interval range into a plurality of second intervals, and counting the number of question-answer pairs in each second interval; obtaining a second maximum value and a second minimum value of the text interval range of the second target interval, obtaining a second theoretical probability according to the second maximum value, the second minimum value, the mean, the variance and the Gaussian distribution function, and obtaining a second actual sample number ratio according to the number of question-answer pairs in the second target interval, the second target interval being any second interval among the plurality of second intervals; adjusting the number of question-answer pairs in the second target interval according to the second theoretical probability and the second actual sample number ratio so that the target sub-sample data set conforms to the spatial distribution of the total sample data set.

[0015] By adopting the above technical solution, the spatial distribution of the sub-sample data set is adjusted to be consistent with the total sample data set, which can ensure that the sub-sample data set can reflect the overall characteristics of the original data set in subsequent analysis or model training, thereby avoiding distortion of analysis results or degradation of model performance due to sample bias. The distribution of key features such as text length in the adjusted target sub-sample data set is more consistent with the Gaussian distribution, and can more comprehensively cover various parts of the original data set, thereby enhancing the representativeness of the data. This helps to discover more valuable information and rules in subsequent data mining, model training and other processes. When the spatial distribution of the sub-sample data set is consistent with the total sample data set, the model trained based on the sub-sample data set will be able to better adapt to the overall characteristics of the original data set, thereby improving the generalization ability of the model. This means that the model can make more accurate and reliable predictions when faced with new and unseen data.

[0016] Optionally, after forming the first multiple rounds of question and answer pairs, the method also includes: determining a fourth preset number based on the total number of question and answer pairs in the question and answer pair data set and the first preset number; randomly selecting a fourth preset number of times from the total sample data set, selecting the first preset number of question and answer pairs each time, to obtain a fourth preset number of sub-sample data sets, and adjusting based on the fourth preset number of sub-sample data sets to obtain a fourth preset number of target sub-sample data sets, forming a fourth preset number of first multiple rounds of question and answer pairs based on the fourth preset number of target sub-sample data sets; forming a second multiple rounds of question and answer pairs based on the fourth preset number of first multiple rounds of question and answer pairs.

[0017] By adopting the above technical solution, multiple sub-sample data sets are generated by random sampling multiple times, and each sub-sample data set is adjusted to obtain a target sub-sample data set, thereby forming multiple first multi-round question-answer pairs. This process increases the diversity of the data set. Using these diverse data sets to train the model can make the model show stronger adaptability and robustness when facing different data inputs. Since each sub-sample data set is obtained through random sampling and adjustment, there are certain differences between them. The existence of these differences enables the models trained based on these sub-sample data sets to better adapt to different types of data inputs, thereby improving the generalization ability of the model. During the training process, multiple sub-sample data sets and target sub-sample data sets can be used for training in parallel, thereby improving the training efficiency. In addition, by comparing the performance of the models trained on different sub-sample data sets, the sensitivity of the data and the stability of the model can be evaluated, thereby optimizing the training process.

[0018] Optionally, determining the fourth preset number according to the total number of question-answer pairs in the question-answer pair data set and the first preset number includes: substituting the total number of question-answer pairs in the question-answer pair data set and the first preset number into a target calculation formula to calculate an initial number; wherein the target calculation formula is as follows:

[0019] ;

[0020] Among them, Epoch is the initial number, N is the total number of question and answer pairs, and K is the first preset number; the initial number is rounded to obtain the fourth preset number.

[0021] By adopting the above technical solution, the target calculation formula can accurately calculate the number of samplings (i.e., the initial number) that should be performed in theory based on the total number of question-answer pair data sets and the first preset number required for each sampling. This process avoids the subjectivity and uncertainty of artificially setting the number of samplings, and ensures the scientificity and rationality of the number of samplings. After determining the initial number, the final fourth preset number is obtained by rounding, which helps to better utilize resources in actual operations. Because in practical applications, the number of samplings must be an integer, the rounding operation ensures this. At the same time, since the number of samplings is accurately calculated based on the total number of data sets and the number of samplings each time, the data set can be maximized to avoid unnecessary waste. By sampling multiple times (the fourth preset number of times), each time the first preset number of question-answer pairs is extracted, multiple different sub-sample data sets can be generated. These sub-sample data sets retain the characteristics of the original data set while also having certain differences. This difference helps to discover more valuable information and laws in subsequent data processing, model training, and other processes, and improve the diversity of data. When the question-answer pair data set is very large, directly processing the entire data set may be very time-consuming and resource-consuming. This method can be used to split a large data set into multiple small data sets (i.e., subsample data sets), and then process these small data sets in parallel. This can not only improve processing efficiency, but also reduce dependence on a single computing resource and support larger-scale data processing.

[0022] In a second aspect of the present application, a system for constructing a large-model multi-round dialogue is provided, including a total sample module, a sub-sample module, an adjustment module and a construction module, wherein: the total sample module is configured to obtain a question-answer pair data set, and adjust the question-answer pair data set based on a Gaussian distribution to obtain a total sample data set; the sub-sample module is configured to determine a first preset number according to a preset batch size, and randomly extract a first preset number of question-answer pairs from the total sample data set to form a sub-sample data set; the adjustment module is configured to adjust the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set; the construction module is configured to form a first multi-round question-answer pair based on the target sub-sample data set.

[0023] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs any of the methods described above.

[0024] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the above methods is executed.

[0025] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0026] 1. By adjusting the question-answering dataset based on Gaussian distribution, the diversity of the dataset can be increased. The application of Gaussian distribution can keep a certain central trend of the data during the adjustment process while introducing an appropriate amount of randomness, which helps the model learn more diverse dialogue patterns and contextual information.

[0027] 2. Generate sub-sample data sets by presetting batch size and randomly extracting them, which ensures that the data used in each training is a random subset of the total sample data set. This strategy not only helps the model see as much different data as possible during training, but also improves data utilization efficiency under limited computing resources;

[0028] 3. Adjusting the sub-sample data set to match the spatial distribution of the total sample data set is a key step to ensure the stability and effectiveness of model training. By maintaining the consistency of data distribution, the model can better learn the true distribution characteristics of the data during training, thereby avoiding overfitting or underfitting problems;

[0029] 4. The first multi-round question-answer pair is formed based on the adjusted target sub-sample dataset. This is the basis for realizing the multi-round dialogue capability of the large model. Multi-round dialogue requires the model to understand and maintain context information. By constructing multi-round question-answer pairs, the model can be trained to maintain the coherence and consistency of the dialogue in consecutive dialogue rounds.

[0030] 5. The entire construction method helps improve the application performance of large models in dialogue systems by finely controlling the diversity and distribution of data and supporting training for multiple rounds of dialogue. The model will be able to understand user intent more accurately and generate more natural and contextual responses, thereby improving the user experience.

[0031] 6. Since a diverse data set adjusted based on Gaussian distribution is used in the training process, the model will be able to show stronger generalization ability when facing unknown or novel dialogue scenarios. This helps the model better adapt to various complex dialogue scenarios in practical applications and improves the robustness and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an overall schematic diagram of the method for constructing a large model multi-round dialogue disclosed in the embodiment of the present application;

[0033] Figure 2 is a schematic diagram of the sample Gaussian distribution disclosed in the embodiment of the present application;

[0034] Figure 3 It is a flowchart of a method for constructing a large model multi-round dialogue disclosed in an embodiment of the present application;

[0035] Figure 4 It is a schematic diagram of the steps for generating a multi-round dialogue of a product disclosed in an embodiment of the present application;

[0036] Figure 5 It is a module diagram of a system for constructing a large model multi-round dialogue disclosed in an embodiment of the present application;

[0037] Figure 6 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0038] Explanation of the reference numerals: 501, total sample module; 502, sub-sample module; 503, adjustment module; 504, construction module; 601, processor; 602, communication bus; 603, user interface; 604, network interface; 605, memory. DETAILED DESCRIPTION

[0039] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0040] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0041] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0042] This embodiment discloses a method for constructing a large model multi-round dialogue, which is applied to a large model training platform. Figure 1 It is a flowchart of the method for constructing a large model multi-round dialogue disclosed in the embodiment of the present application, such as Figure 1 As shown, the construction method includes the following steps:

[0043] S110, obtaining a question-answer pair dataset, and adjusting the question-answer pair dataset based on Gaussian distribution to obtain a total sample dataset;

[0044] Collect question-answer pair datasets from various sources. These datasets should contain multiple question-answer pairs, each consisting of a question and an answer. For each question-answer pair in the dataset, calculate its total text length. This is usually obtained by adding the text length of the question and the answer. The text length can be defined by counting the number of characters, words, or bytes in a string, depending on the needs of the application scenario. Count the text length of each question-answer pair calculated and plot it into a histogram or frequency distribution graph. The horizontal axis represents the text length, and the vertical axis represents the number of question-answer pairs with that length. Observe the distribution graph to determine whether it is close to a Gaussian distribution. A Gaussian distribution is usually high in the middle and low on both sides, and is symmetrical. If the distribution of the dataset is significantly different from a Gaussian distribution, it may need to be adjusted. Figure 2is a schematic diagram of the sample Gaussian distribution disclosed in the embodiment of the present application, such as Figure 2 As shown in Figure 2, although the samples generally show a Gaussian distribution, they may not satisfy the Gaussian distribution locally, such as Figure 2 The number of samples in part 201 can be appropriately reduced, and the number of samples in part 202 can be appropriately increased, so that the local area can also satisfy the Gaussian distribution.

[0045] Adjusting the question-answer pair data set based on Gaussian distribution to obtain a total sample data set includes: dividing the question-answer pair data set into multiple first intervals, counting the first theoretical probability and the first actual sample number ratio of the first target interval, adjusting the number of question-answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and forming a total sample data set based on all adjusted question-answer pairs in the first target interval, the first target interval being any first interval among the multiple first intervals.

[0046] The question-answer pairs in the question-answer pair dataset are divided into multiple first intervals according to their total text length. The division of these intervals can be set according to the specific situation and needs of the data, such as dividing according to equal text length intervals, or dividing according to a preset target number of intervals. Each interval represents a range of text length. For each divided first interval, the following statistics need to be performed:

[0047] First theoretical probability: Calculate the theoretical probability corresponding to the text length in the first interval based on Gaussian distribution. This usually involves the probability density function of Gaussian distribution, through which the probability of values ​​in any interval can be calculated. First actual sample number ratio: Count the proportion of question and answer pairs in the actual question and answer pair data set that fall within this interval to the total number of question and answer pairs. According to the difference between the first theoretical probability and the first actual sample number ratio, adjust the number of question and answer pairs in the first target interval. The goal of the adjustment is to make the actual sample number ratio of each interval as close to its theoretical probability as possible, so that the distribution of the overall data set is closer to the Gaussian distribution. If the first actual sample number ratio is higher than the first theoretical probability, it means that there are too many samples in this interval, and the number of question and answer pairs in this interval needs to be reduced. This can be achieved by randomly deleting some question and answer pairs, or by using more complex resampling strategies to maintain data diversity. If the first actual sample number ratio is lower than the first theoretical probability, it means that there are not enough samples in this interval, and the number of question and answer pairs in this interval needs to be increased. This can be achieved by copying existing question and answer pairs (pay attention to avoid the risk of overfitting), generating synthetic data (such as using a text generation model), or migrating data from other similar datasets. Repeat the above statistical and adjustment process for all the first intervals. Each interval needs to be adjusted independently according to the difference between its theoretical probability and the actual sample number. After all intervals have been adjusted, all the adjusted question-answer pairs are combined to form a new total sample data set. This data set should be closer to the Gaussian distribution in terms of text length distribution.

[0048] By counting the number of question-answer pairs in each first interval and comparing it with the theoretical probability calculated based on the Gaussian distribution, the intervals in the sample distribution that do not conform to the Gaussian distribution can be identified. Subsequently, by adjusting the number of question-answer pairs in these intervals, the distribution of the entire data set is made closer to the ideal Gaussian distribution. Sample sets that conform to the Gaussian distribution help improve the generalization ability and prediction accuracy of machine learning models. When training on a data set that does not conform to the Gaussian distribution, the model may be biased towards majority class samples, resulting in a decrease in the prediction performance of minority class samples. The above adjustment method can reduce this bias, so that the model can treat all data points more fairly during the training process, thereby improving the overall performance and robustness of the model. A data set that conforms to the Gaussian distribution means that the model is exposed to various possible situations during the training process, which helps the model learn more comprehensive and accurate knowledge. Therefore, a model trained using a data set adjusted by the Gaussian distribution can give more stable and reliable prediction results when facing unknown data.

[0049] The question-answer pair data set is divided into multiple first intervals, the first theoretical probability and the first actual sample number ratio of the first target interval are counted, the number of question-answer pairs in the first target interval is adjusted according to the first theoretical probability and the first actual sample number ratio, and the total sample data set is formed according to all the adjusted question-answer pairs in the first target interval, including: counting the text lengths of the question-answer pairs in the question-answer pair data set to form a text length set, determining the first text interval range according to the first text length and the second text length, the first text length being the minimum text length in the text length set, and the second text length being the maximum text length in the text length set; calculating the mean and square of the text length in the text length set. difference, evenly divide the first text interval range into multiple first intervals, divide the question and answer pairs in the question and answer pair data set into corresponding first intervals according to the text length, and count the number of question and answer pairs in each first interval; obtain the first maximum value and the first minimum value of the text interval range of the first target interval, obtain the first theoretical probability according to the first maximum value, the first minimum value, the mean, the variance and the Gaussian distribution function, and obtain the first actual sample number ratio according to the number of question and answer pairs in the first target interval; adjust the number of question and answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and form a total sample data set according to all the adjusted question and answer pairs in the first target interval.

[0050] The text lengths of all question-answer pairs in the question-answer pair data set are counted to form a text length set. The minimum text length (first text length) and the maximum text length (second text length) are found from this set. The first text length and the second text length determine the interval range of the text length (first text interval range). For example, the first text length is 10 and the second text length is 110. The mean and variance of all text lengths in the text length set are calculated. The first text interval range is evenly divided into multiple first intervals. The multiple first intervals can be any number and can be specifically set according to needs. For example, in the embodiment of the present application, they are divided into 100 intervals, and the length of each interval is 1. According to the text length of each question-answer pair, they are divided into corresponding first intervals, and the number of question-answer pairs in each interval is counted. This step is the key to understanding the current sample distribution. For any first target interval (i.e., any one of the multiple first intervals), the first maximum value and the first minimum value of the text interval range of the first target interval are determined. For example, the text interval range of the first first interval is [10, 11). For the convenience of calculation, the first maximum value is 11 and the first minimum value is 10. The first maximum value, the first minimum value, the mean, the variance, and the Gaussian distribution function (usually the probability density function of the normal distribution) are used to calculate the theoretical probability of the interval (the first theoretical probability). At the same time, based on the number of question-answer pairs in the first target interval, the actual sample ratio (the first actual sample ratio) is calculated. The first theoretical probability and the first actual sample ratio are compared.

[0051] Specifically, the Gaussian distribution function f(x) is as follows:

[0052] ;

[0053] in, represents the mean, Represents variance.

[0054] Integrate the Gaussian distribution function to obtain the first theoretical probability, with the upper limit of the integral being the first maximum value and the lower limit of the integral being the first minimum value. If the actual number of samples accounts for a greater proportion than the theoretical probability, negative sampling is performed, i.e., some samples are randomly reduced from this interval until the error between the two is less than a threshold value (e.g., 1%). On the contrary, if the actual number of samples accounts for a smaller proportion than the theoretical probability, positive sampling is performed, i.e., samples from this interval are randomly selected for replication to increase the number of samples, again until the error is less than the threshold value. Repeat the above adjustment process until the number of question-answer pairs in all first target intervals has been adjusted, so that the sample distribution of the entire data set is closer to a Gaussian distribution. Finally, all adjusted question-answer pairs in the first target interval are combined to form a total sample data set.

[0055] Adjusting the number of question and answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio includes: comparing the first theoretical probability with the first actual sample number ratio; when the first theoretical probability is greater than the first actual sample number ratio, calculating a second preset number, randomly selecting a second preset number of samples from the first target interval for copying and updating the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than a first threshold; when the first theoretical probability is less than the first actual sample number ratio, calculating a third preset number, randomly taking a third preset number of samples from the first target interval and updating the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than the first threshold.

[0056] Compare the first theoretical probability with the first actual sample number ratio. This comparison is to determine whether the number of samples in the current interval is too many or too few, so as to make corresponding adjustments. When the first theoretical probability is greater than the first actual sample number ratio, it means that the number of samples in the current interval (the first target interval) is too few, and the number of samples needs to be increased to approach the theoretical probability. At this time, calculate the second preset number: the second preset number is calculated based on the difference between the first theoretical probability and the first actual sample number and a certain adjustment strategy (such as proportional increase, fixed number increase, etc.). The purpose is to determine the number of samples that need to be copied from outside the current interval (or the entire data set) to the current interval, or the number of existing samples copied in the current interval. Randomly select the second preset number of samples from the current interval for copying, and add these copied samples to the current interval. Note that you can also choose to copy samples from other similar intervals or the entire data set, depending on the implementation details and the characteristics of the data set. Due to the increase in the number of samples, the first theoretical probability and the first actual sample number ratio of the current interval need to be recalculated and updated. The above process is repeated until the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than a preset first threshold (eg, 1%), at which time it is considered that the number of samples in the current interval is close enough to the theoretical probability.

[0057] When the first theoretical probability is less than the first actual sample number ratio, it means that the number of samples in the current interval is too large, and the number of samples needs to be reduced to approach the theoretical probability. At this time, calculate the third preset number: similar to the calculation of the second preset number, the third preset number is calculated based on the difference between the first theoretical probability and the first actual sample number ratio and a certain adjustment strategy. The purpose is to determine the number of samples that need to be removed from the current interval. Randomly take out the third preset number of samples from the current interval, and remove the selected samples from the data set (or mark them as invalid). Similarly, the first theoretical probability and the first actual sample number ratio of the current interval need to be recalculated and updated. Repeat the above process until the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than the first threshold.

[0058] The above has explained how to divide the intervals according to equal text length intervals. The following describes the specific steps of dividing the intervals according to the preset target number of intervals:

[0059] (1) Preset the number of preset target intervals you want to divide, for example, the number of preset target intervals is 8;

[0060] (2) Based on the Gaussian distribution characteristics, the total number of samples, and the preset number of target intervals, determine the preset number of samples for each interval (the interval lengths are different, and the number of samples in each interval is different): for example, the total number of samples is 200, and the preset number of target intervals is 8. Then the preset number of samples for each of the 8 intervals from left to right can be 5, 18, 35, 42, 42, 35, 18, and 5.

[0061] (3) For each interval, the interval division (the start and end text lengths of the interval) is determined based on the preset sample number of the interval and the actual sample size of each text length interval. For example, for the first interval from left to right, the preset sample number is 5. If the actual sample size of the text length interval (0, 9] is 2 and the actual sample size of the text length interval (0, 10] is 6, then the text range of the first interval is determined to be (0, 10). For each interval, according to this division method, the preset sample number of the interval and the actual sample size of each text length interval are compared so that the actual sample size is approximately equal to the preset sample number, and the preset target number of text length intervals is obtained.

[0062] The method of calculating the first theoretical probability and the first actual sample number ratio and making adjustments based on the first theoretical probability and the first actual sample number ratio is similar to the method of dividing the intervals according to equal text length intervals, and will not be repeated here.

[0063] S120, determining a first preset number according to a preset batch size, and randomly selecting a first preset number of question-answer pairs from the total sample data set to form a sub-sample data set;

[0064] According to the requirements of the training task, the limitations of computing resources (such as GPU memory), and training efficiency, a suitable batch size is preset, and then the number of question-answer pairs selected each time (i.e., the first preset number) is determined based on the batch size. Use a random number generator to randomly extract the first preset number of question-answer pairs from the total sample data set. This step ensures the diversity and randomness of the sub-sample data set, which helps the model learn the overall distribution characteristics of the data during training. The first preset number of randomly selected question-answer pairs are organized to form a sub-sample data set, which will be used for one iteration of model training.

[0065] S130, adjusting the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set;

[0066] Spatial distribution refers to the interval of the first interval and the number of samples in each first interval. If the spatial distribution of the sub-sample dataset is consistent with the spatial distribution of the total sample dataset, there is no need to adjust the sub-sample dataset. If the spatial distribution of the sub-sample dataset is inconsistent with the spatial distribution of the total sample dataset, the sub-sample dataset needs to be adjusted to obtain the target sub-sample dataset. The spatial distribution of the target sub-sample dataset is consistent with the spatial distribution of the total sample dataset.

[0067] The sub-sample data set is adjusted to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set, including: counting the text lengths of the question-answer pairs in the sub-sample data set to form a sub-text length set, determining the second text interval range according to the third text length and the fourth text length, and adjusting the second text interval range according to the mean and the variance to obtain a third text interval range, wherein the third text length is the minimum text length in the sub-text length set, and the fourth text length is the maximum text length in the sub-text length set; evenly dividing the third text interval range into a plurality of second intervals, and counting the number of question-answer pairs in each second interval; obtaining a second maximum value and a second minimum value of the text interval range of the second target interval, obtaining a second theoretical probability according to the second maximum value, the second minimum value, the mean, the variance and the Gaussian distribution function, and obtaining a second actual sample number ratio according to the number of question-answer pairs in the second target interval, wherein the second target interval is any second interval among the plurality of second intervals; adjusting the number of question-answer pairs in the second target interval according to the second theoretical probability and the second actual sample number ratio so that the target sub-sample data set conforms to the spatial distribution of the total sample data set.

[0068] The text length of each question-answer pair is counted from the sub-sample data set to form a sub-text length set. An initial text interval range (i.e., the second text interval range) is determined based on the minimum text length (the third text length) and the maximum text length (the fourth text length) in the sub-text length set. The second text interval range is adjusted using the text length mean and variance of the total sample data set. Specifically, the adjusted interval range (the third text interval range) is [min(min_len, -3σ), max(max_len, +3σ)], where min_len represents the third text length, and max_len represents the fourth text length. Represents the mean, σ represents the variance, min() represents the minimum value, and max() represents the maximum value. Divide the adjusted third text interval range into multiple second intervals (for example, 50 intervals) on average. Then, count the number of question-answer pairs contained in each second interval, that is, the actual number of samples in each interval. For each second interval (that is, the second target interval), calculate the theoretical probability (second theoretical probability) in the interval based on the second maximum value and second minimum value of the second target interval, the mean and variance of the total sample data set, and the Gaussian distribution function. At the same time, calculate the actual sample number ratio (second actual sample number ratio) of the interval based on the actual number of question-answer pairs in the interval. According to the comparison result between the second theoretical probability and the second actual sample ratio, the number of samples in each second target interval is adjusted: if the second actual sample ratio is greater than the second theoretical probability, negative sampling is performed to randomly reduce some samples from the interval until the error between the second actual sample ratio and the second theoretical probability is less than the third threshold (for example, 1%); if the second actual sample ratio is less than the second theoretical probability, positive sampling is performed to randomly select samples in the interval for replication until the error between the second actual sample ratio and the second theoretical probability is less than 1%. Through the above steps, the number of samples in each interval in the sub-sample data set is adjusted, and the target sub-sample data set finally obtained will be closer to the spatial distribution of the total sample data set in terms of text length distribution. Such adjustments help improve the effect and generalization ability of subsequent model training.

[0069] S140: Based on the target sub-sample data set, a first multi-round question-answer pair is formed.

[0070] Select appropriate question-answer pairs from the target subsample dataset to construct the dialogue logic between the question-answer pairs. The dialogue logic should conform to the habits and rules of natural language communication to ensure the fluency and coherence of the dialogue. In multi-round question-answer dialogues, the management of context information is crucial. It is necessary to ensure that each round of question-answer can correctly understand and utilize the context information of the previous round of question-answer. This can be achieved by embedding specific identifiers in the question-answer pairs, using context vectors or memory networks, and other technologies. The quality of context management directly affects the accuracy and efficiency of the dialogue. Combine the selected question-answer pairs according to the constructed dialogue logic to form a complete multi-round question-answer dialogue. In the combination process, it is necessary to pay attention to the connection and transition between the question-answer pairs to ensure the naturalness and fluency of the dialogue. At the same time, the question-answer pairs need to be formatted, such as removing redundant information and adjusting sentence structure, to improve the readability and comprehensibility of the dialogue.

[0071] In other embodiments, constructing the dialogue logic between question-answer pairs includes determining the order of question-answer pairs, setting the starting point of the dialogue, designing the turning point of the dialogue, and handling abnormal situations in the dialogue. For example,

[0072] User input: Hello, I want to travel from Beijing to Shanghai, can you give me some advice?

[0073] Possible platform response: Hello, we are glad to help you plan your trip from Shanghai to Beijing. Would you like to know about transportation, recommended attractions or accommodation information?

[0074] The platform guides the start of the conversation by asking users about their specific needs, while also clarifying the starting and ending points of the trip.

[0075] The following are possible conversation formats:

[0076] User: I would like to know what transportation options are available.

[0077] Platform: From Shanghai to Beijing, you can choose to take the high-speed rail, plane or drive. The high-speed rail takes about 5 hours, and the plane takes about 2 hours. Self-driving needs to be arranged according to road conditions and rest time.

[0078] User: So how much is the high-speed rail ticket?

[0079] Platform: High-speed rail ticket prices vary depending on seat type and ticket purchase time, generally ranging from a few hundred yuan to thousands of yuan. You can check the specific price on the ticket purchase website.

[0080] User (Turning Point): Okay, then I would like to know more about the famous attractions in Beijing.

[0081] Platform: Of course. Beijing has many famous attractions, such as the Forbidden City, Tiananmen Square, the Great Wall, etc. Each attraction has its own unique charm and historical and cultural background.

[0082] At this turning point, the user changed from asking about transportation to asking about tourist attractions, and the system flexibly adjusted the topic of the conversation based on the user's input.

[0083] User (abnormal situation): What’s the weather like in Beijing? I suddenly remembered this is very important.

[0084] Platform: The weather in Beijing varies with the season. Now it is autumn, and it is warm and pleasant during the day and slightly cool at night. I suggest you check the latest weather forecast to better plan your trip.

[0085] During the conversation, users may ask some unexpected questions. The platform needs to be able to identify and properly handle these abnormal situations to keep the conversation smooth.

[0086] After forming the first multiple rounds of question-answer pairs, the method also includes: determining a fourth preset number based on the total number of question-answer pairs in the question-answer pair data set and the first preset number; randomly extracting a fourth preset number of times from the total sample data set, extracting the first preset number of question-answer pairs each time, obtaining a fourth preset number of sub-sample data sets, and adjusting the fourth preset number of target sub-sample data sets based on the fourth preset number of sub-sample data sets to obtain a fourth preset number of target sub-sample data sets, forming a fourth preset number of first multiple rounds of question-answer pairs based on the fourth preset number of target sub-sample data sets; forming a second multiple rounds of question-answer pairs based on the fourth preset number of first multiple rounds of question-answer pairs.

[0087] According to the total number of question-answer pair data sets and the first preset number, the fourth preset number is determined. The fourth preset number indicates the number of random samplings from the total sample data set, and each sampling will obtain a new sub-sample data set. The first preset number specifies the number of question-answer pairs to be selected from the total sample data set at each sampling. Randomly extract a total of the fourth preset number of question-answer pairs from the total sample data set. Each extraction is performed according to the first preset number, that is, a fixed number of question-answer pairs are extracted each time to form a sub-sample data set. For each sub-sample data set obtained by extraction, adjustments are made according to the method described above, including counting text length, adjusting interval range, dividing intervals, counting sample numbers, calculating theoretical probabilities and actual proportions, and performing positive or negative sampling of sample numbers to obtain a target sub-sample data set that conforms to the spatial distribution of the total sample data set. This process will be repeated for the fourth preset number of times, so that a fourth preset number of target sub-sample data sets will be obtained. Based on each target sub-sample data set, the first multiple rounds of question-answer pairs are formed according to the method described above. Therefore, a fourth preset number of first multi-round question-answer pairs will eventually be obtained, and each multi-round question-answer pair is generated based on an adjusted target sub-sample data set. According to the fourth preset number of first multi-round question-answer pairs, a second multi-round question-answer pair is formed in a certain way (such as merging, selecting, weighting, etc.). The specific implementation method of this process depends on the specific application scenario and requirements. For example, all first multi-round question-answer pairs can be directly merged into a large set of question-answer pairs, or some of the highest quality or most representative question-answer pairs can be selected from them to form a second multi-round question-answer pair. Alternatively, a weighted merger can be performed based on the contribution or importance of each first multi-round question-answer pair.

[0088] Determining a fourth preset number according to the total number of question-answer pairs in the question-answer pair data set and the first preset number includes:

[0089] Substitute the total number of question-answer pairs in the question-answer pair data set and the first preset number into the target calculation formula to calculate the initial number; wherein the target calculation formula is as follows:

[0090] ;

[0091] Among them, Epoch is the initial number, N is the total number of question and answer pairs, and K is the first preset number; the initial number is rounded to obtain the fourth preset number.

[0092] In the embodiment of the present application, N is the total number of question-answer pairs, and K is the first preset number, that is, there are N question-answer pairs in total, and K question-answer pairs are drawn each time, and the drawn question-answer pairs are put back, so the probability of each question-answer pair being drawn is K / N. (N / K)*(K / N)=1, that is, drawing N / K times can ensure that each question-answer pair can be drawn. It is a redundant part, which ensures that every sample can be drawn again. Since the calculation result may be a decimal, and the number of samples or the number of processing times must be an integer, the initial number needs to be rounded up. This is usually rounded up (that is, taking the smallest integer greater than the initial number) to ensure that the result is a feasible integer. For example, if N is 100 and K is 10, then Epoch=100 / 10+2=12. Since 12 is an integer, the fourth preset number is 12.

[0093] The forming of the second multi-round question-answer pairs according to the fourth preset number of the first multi-round question-answer pairs includes: extracting keywords of the question-answer pair text in the first target sub-sample data set, determining the topic of the first target sub-sample data set according to the keywords, the first target sub-sample data set being any one of the fourth preset number of the target sub-sample data sets; wherein one target sub-sample data set corresponds to one of the first multi-round question-answer pairs; respectively calculating the relevance values ​​of the fourth preset number of the topics and the subject corresponding to the question input by the user, taking the second target sub-sample data set corresponding to the maximum relevance value as the first target sub-sample data set, and sorting the fourth preset number of the target sub-sample data sets according to the topic similarity index; selecting the first question-answer pair of the first target sub-sample data set as the starting point of the conversation, and splicing the subsequent target sub-sample data sets according to the semantic similarity to form the second multi-round question-answer pairs.

[0094] For each target sub-sample dataset, the platform will extract keywords from the question-answer text. These keywords can be extracted based on natural language processing models such as TF-IDF, Word2Vec, and BERT, and they can reflect the main content of the question-answer pair. Then, based on these keywords, the platform can further determine the topic of the target sub-sample dataset. Topics can be pre-defined classification labels or dynamically generated by clustering algorithms. The platform will calculate the relevance values ​​between multiple topics and the topic corresponding to the question entered by the user. This relevance can be calculated using metrics such as cosine similarity and Jaccard similarity. The specific selection method can depend on the characteristics of the data and the design of the platform. After the calculation is completed, the platform will use the target sub-sample dataset corresponding to the maximum relevance value as the first target sub-sample dataset, that is, it is considered that this dataset is most relevant to the user's question. Subsequently, the platform will sort the remaining target sub-sample datasets according to the similarity of the topics. The basis for sorting can be the similarity score between topics or other indicators that can reflect the relationship between topics. The purpose of sorting is to organize the question-answer pairs in a logical order (such as the progression and transition of topics) when multiple rounds of question-answer pairs are formed in the future. After determining the first target sub-sample dataset, the platform will select the first question-answer pair of the dataset as the starting point of the conversation. This question-answer pair is usually considered to be the most directly related to the user's question or the most interesting to the user. Then, the platform will splice the subsequent target sub-sample datasets according to semantic similarity to form multiple rounds of question-answer pairs. The semantic similarity here can be obtained by calculating the text similarity between question-answer pairs, or a more complex semantic matching model can be used. During the splicing process, the platform will try to maintain the logical coherence and contextual consistency between question-answer pairs to ensure that the formed multiple rounds of question-answer pairs conform to the habits and rules of natural language communication.

[0095] By extracting keywords from the text of the question-answer pairs in the target subsample dataset and determining the topic accordingly, it is possible to ensure that each subsample dataset revolves around a clear theme or topic. This focus helps to improve the pertinence and relevance of the question-answer pairs. By calculating the relevance of multiple topics to the topic corresponding to the user's input question and selecting the subsample dataset corresponding to the maximum relevance as the first target subsample dataset, the matching degree between the question-answer pairs and the user's needs is further enhanced, and the practicality and effectiveness of the conversation are improved. By sorting multiple target subsample datasets according to the similarity of the topics, it is possible to ensure that when forming multiple rounds of question-answer pairs, adjacent question-answer pairs maintain a certain degree of coherence and progressiveness in terms of topics, thereby improving the natural fluency of the conversation. By selecting the first question-answer pair of the first target subsample dataset as the starting point of the conversation and splicing the subsequent target subsample datasets according to the semantic similarity, it is possible to ensure that the entire conversation process is logically consistent and coherent, thereby enhancing the user experience. In addition, through the above second multi-round question-answer pair formation method, more sample variants and topic coverage can be introduced, thereby increasing the diversity and generalization ability of the question-answer pairs. This enables the dialogue system to handle a wider range of questions and scenarios, improving the flexibility and adaptability of the system.

[0096] The splicing of subsequent target sub-sample data sets according to semantic similarity to form a second multi-round question-answer pair includes: using a pre-trained model to convert the text in the question-answer pairs in all target sub-sample data sets into text vectors, and calculating the semantic similarity between the first question in the next target sub-sample data set in the sorting and the last answer in the previous target sub-sample data set based on the text vector; when the semantic similarity is greater than a second threshold, merging the first question in the next target sub-sample data set with the last answer in the previous target sub-sample data set; when the semantic similarity is less than or equal to the second threshold, inserting transitional text between the first question in the next target sub-sample data set and the last answer in the previous target sub-sample data set, and merging the first question in the next target sub-sample data set with the last answer in the previous target sub-sample data set according to the transitional text.

[0097] The texts in the question-answer pairs in all target subsample datasets are converted into text vectors using pre-trained models (such as BERT, Word2Vec, etc.). A text vector is a numerical representation of text that can capture the semantic information of the text. This process enables computers to understand and compare the semantic relationship between different texts. Based on the converted text vectors, the semantic similarity between the first question in the next target subsample dataset and the last answer in the previous target subsample dataset in the sorting is calculated. Semantic similarity is an indicator that measures the degree of semantic similarity or proximity between two texts. Common calculation methods include cosine similarity, Euclidean distance, etc. When the calculated semantic similarity is greater than the second threshold, the two texts are considered to be semantically close enough and can be directly merged. That is, the first question in the next target subsample dataset is regarded as a continuation or response to the last answer in the previous target subsample dataset, forming a natural conversation flow. When the semantic similarity is less than or equal to the second threshold, the two texts are considered not semantically close enough, and direct merging may result in abrupt or difficult to understand conversations. At this time, a transitional text is inserted between the two to smoothly connect the two texts. Transitional text can be automatically generated based on the context, or it can be a preset template or phrase. Its purpose is to maintain the coherence and fluency of the conversation. Based on the above decision, the first question in the next target subsample dataset is merged with the last answer in the previous target subsample dataset. If the semantic similarity is high enough, they are directly merged; if not, they are merged after inserting transitional text. This process is repeated until all target subsample datasets are reasonably spliced ​​into multiple rounds of question-answer pairs.

[0098] By using a pre-trained model to convert text into text vectors and calculating the semantic similarity between different question-answer pairs based on these text vectors, it is possible to ensure that the previous and subsequent question-answer pairs remain semantically coherent when splicing question-answer pairs. This splicing method based on semantic similarity can better reflect the natural fluency of the conversation than simple sequential splicing. When the semantic similarity is greater than the second threshold, the first question in the next target subsample dataset is directly merged with the last answer in the previous target subsample dataset. This method is simple and direct and is suitable for question-answer pairs with high semantic correlation. When the semantic similarity is less than or equal to the second threshold, the question-answer pairs are merged by inserting transitional text, which increases the flexibility and adaptability of the conversation and enables a smooth transition even between question-answer pairs that are not completely related. Inserting transitional text not only helps to improve the naturalness and coherence of the conversation, but also provides users with better understanding and acceptance. The transitional text can be a summary of the previous answer, a guide to the next question, or a bridge between the two, thereby helping users better understand the process and logic of the conversation. The entire splicing process is based on pre-trained models and automated calculation methods, without manual intervention, which greatly improves the efficiency and accuracy of the processing. At the same time, since the calculation of text vectors and the judgment of semantic similarity can be quickly completed on the computer, multiple rounds of question-answer pairs can be generated in real time or near real time to meet the needs of practical applications.

[0099] Figure 3 It is a flowchart of the method for constructing a large model multi-round dialogue disclosed in the embodiment of the present application, such as Figure 3 As shown, the question-answer data set is loaded, and Gaussian distribution calculation is performed, and the configuration is loaded. The configuration includes batch size and round threshold (M, which is the fourth preset number above). When the iteration round is less than the round threshold, the data set is shuffled, and a subset is extracted. The Gaussian distribution calculation is used to determine whether the distribution of the subset is abnormal. When the distribution of the subset is abnormal, the subset is corrected. If the distribution of the subset is normal, the subset is spliced, and the iteration round is increased by one. When the iteration round is not less than the fourth preset number, it ends.

[0100] Figure 4 Schematic diagram of the steps for generating a multi-round dialogue of a product disclosed in the embodiment of the present application, such as Figure 4 As shown, the question-answer pair set includes A, B, C, D, E, F, G and H. A subset is randomly drawn, including B, D, E and G. Gaussian check is performed on the subset to remove the unreasonable question-answer pair E. The corrected subset includes question-answer pairs B, D and G. The question-answer pairs in the subset are then merged to form a multi-round dialogue BDG, and multiple multi-round dialogues are spliced ​​to form a multi-round dialogue set.

[0101] Figure 3The embodiments of the present application are described in detail. Figure 4 The embodiments of the present application are briefly described. Figure 3 and Figure 4 The description can provide a deeper understanding of the embodiments of the present application. The embodiments of the present application mainly use the two indicators BLUE and ROUGE to evaluate the effect of the fine-tuning of the model. Compared with the other two technologies, both have increased model improvements. From the perspective of Blue-4 alone, compared with the manual strategy combination construction method, the embodiment of the present application has a relative improvement of 153.32%; compared with the large model generation construction method, the embodiment of the present application has a relative improvement of 107.43%. BLEU (Bilingual Evaluation Understudy) is an indicator for evaluating machine-translated texts, but it can also be used to evaluate the quality of generated texts by comparing the degree of overlap between machine-generated texts and human-generated reference texts. ROUGE (Recall-Oriented Understudy for GistingEvaluation) can be seen as an improved version of BLEU, focusing on recall rather than precision. In other words, it looks at the number of n-grams in the reference translation that appear in the output. The experimental results are shown in the following table:

[0102] ;

[0103] BLUE-4 measures the n-tuple (here 4-tuple) matching between the generated text and the reference text. From the perspective of BLUE-4, the embodiment of the present application has greatly improved the quality of the generated text compared to the prior art. ROUGE-1, ROUGE-2 and ROUGE-L evaluate the recall rate of the generated text from the perspectives of unigrams, bigrams and longest common subsequences, respectively. In terms of these indicators, the embodiment of the present application also performs well, outperforming the other two methods, proving its effectiveness in generating multi-round dialogues.

[0104] This embodiment also discloses a system for constructing a large model multi-round dialogue. Figure 5 Schematic diagram of a module of a system for constructing a large model multi-round dialogue disclosed in an embodiment of the present application. Figure 5As shown, the system includes a total sample module 501, a sub-sample module 502, an adjustment module 503 and a construction module 504, wherein: the total sample module 501 is configured to obtain a question-answer pair data set, and adjust the question-answer pair data set based on a Gaussian distribution to obtain a total sample data set; the sub-sample module 502 is configured to determine a first preset number according to a preset batch size, and randomly extract a first preset number of question-answer pairs from the total sample data set to form a sub-sample data set; the adjustment module 503 is configured to adjust the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set; the construction module 504 is configured to form a first multi-round question-answer pair based on the target sub-sample data set.

[0105] Optionally, the total sample module 501 is configured to: divide the question-answer pair data set into multiple first intervals, count the first theoretical probability and the first actual sample number ratio of the first target interval, adjust the number of question-answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and form a total sample data set based on all adjusted question-answer pairs in the first target interval, where the first target interval is any first interval among the multiple first intervals.

[0106] Optionally, the total sample module 501 is configured to: count the text lengths of the question and answer pairs in the question and answer pair data set to form a text length set, determine the first text interval range according to the first text length and the second text length, the first text length is the minimum text length in the text length set, and the second text length is the maximum text length in the text length set; calculate the mean and variance of the text length in the text length set, divide the first text interval range into multiple first intervals on average, divide the question and answer pairs in the question and answer pair data set into corresponding first intervals according to the text length, and count the number of question and answer pairs in each first interval; obtain the first maximum value and the first minimum value of the text interval range of the first target interval, obtain the first theoretical probability according to the first maximum value, the first minimum value, the mean, the variance and the Gaussian distribution function, and obtain the first actual sample number ratio according to the number of question and answer pairs in the first target interval; adjust the number of question and answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and form a total sample data set according to all the adjusted question and answer pairs in the first target interval.

[0107] Optionally, the total sample module 501 is configured to: compare the first theoretical probability with the first actual sample number ratio; when the first theoretical probability is greater than the first actual sample number ratio, calculate the second preset number, randomly select the second preset number of samples from the first target interval for copying and update the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than a first threshold value; when the first theoretical probability is less than the first actual sample number ratio, calculate the third preset number, randomly take out the third preset number of samples from the first target interval and update the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than the first threshold value.

[0108] Optionally, the adjustment module 503 is configured to: count the text lengths of the question-answer pairs in the sub-sample data set to form a sub-text length set, determine the second text interval range according to the third text length and the fourth text length, and adjust the second text interval range according to the mean and variance to obtain a third text interval range, the third text length is the minimum text length in the sub-text length set, and the fourth text length is the maximum text length in the sub-text length set; divide the third text interval range into multiple second intervals on average, and count the number of question-answer pairs in each second interval; obtain the second maximum and second minimum values ​​of the text interval range of the second target interval, obtain the second theoretical probability according to the second maximum, second minimum, mean, variance and Gaussian distribution function, obtain the second actual sample number ratio according to the number of question-answer pairs in the second target interval, the second target interval is any second interval among the multiple second intervals; adjust the number of question-answer pairs in the second target interval according to the second theoretical probability and the second actual sample number ratio so that the target sub-sample data set conforms to the spatial distribution of the total sample data set.

[0109] Optionally, the system also includes an execution module, which is configured to: determine a fourth preset number based on the total number of question and answer pairs in the question and answer pair data set and the first preset number; randomly select a fourth preset number of times from the total sample data set, and select the first preset number of question and answer pairs each time to obtain a fourth preset number of sub-sample data sets, and adjust the fourth preset number of target sub-sample data sets based on the fourth preset number of sub-sample data sets to obtain a fourth preset number of target sub-sample data sets, and form a fourth preset number of first multi-round question and answer pairs based on the fourth preset number of target sub-sample data sets; and form a second multi-round question and answer pairs based on the fourth preset number of first multi-round question and answer pairs.

[0110] Optionally, the execution module is configured to: substitute the total number of question-answer pairs in the question-answer pair data set and the first preset number into the target calculation formula to calculate an initial number; wherein the target calculation formula is as follows:

[0111] ;

[0112] Among them, Epoch is the initial number, N is the total number of question and answer pairs, and K is the first preset number; the initial number is rounded to obtain the fourth preset number.

[0113] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0114] This embodiment also discloses an electronic device, referring to Figure 6 The electronic device may include: at least one processor 601 , at least one communication bus 602 , a user interface 603 , a network interface 604 , and at least one memory 605 .

[0115] The communication bus 602 is used to realize the connection and communication between these components.

[0116] The user interface 603 may include a display screen (Display) and a camera (Camera). The optional user interface 603 may also include a standard wired interface and a wireless interface.

[0117] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0118] Among them, the processor 601 may include one or more processing cores. The processor 601 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling data stored in the memory 605. Optionally, the processor 601 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 601 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 601, and it can be implemented separately through a chip.

[0119] Among them, the memory 605 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 605 includes a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 605 may optionally be at least one storage device located away from the aforementioned processor 601. As Figure 6 As shown, the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a method for constructing a large model multi-round dialogue.

[0120] exist Figure 6In the electronic device shown, the user interface 603 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 601 can be used to call the application program for constructing a method for constructing a large model multi-round dialogue stored in the memory 605. When executed by one or more processors 601, the electronic device executes one or more methods in the above-mentioned embodiments.

[0121] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0122] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 605, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 605 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0127] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for constructing a large-model multi-round dialogue, characterized in that: Applied to a large model training platform, the construction method includes: Obtain a question-answer pair dataset, and adjust the question-answer pair dataset based on Gaussian distribution to obtain a total sample dataset; Determine a first preset number according to a preset batch size, and randomly select the first preset number of question-answer pairs from the total sample data set to form a sub-sample data set; Adjusting the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set; Based on the target sub-sample data set, a first multi-round question-answer pair is formed. The step of adjusting the question-answer pair data set based on Gaussian distribution to obtain a total sample data set includes: The question-answer pair data set is divided into multiple first intervals, and a first theoretical probability and a first actual sample number ratio of a first target interval are counted. The number of question-answer pairs in the first target interval is adjusted according to the first theoretical probability and the first actual sample number ratio. A total sample data set is formed according to all the adjusted question-answer pairs in the first target interval, where the first target interval is any first interval among the multiple first intervals. The step of dividing the question-answer pair data set into a plurality of first intervals, counting a first theoretical probability and a first actual sample number ratio of a first target interval, adjusting the number of question-answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and forming a total sample data set according to all adjusted question-answer pairs in the first target interval includes: Counting the text lengths of the question-answer pairs in the question-answer pair data set to form a text length set, and determining a first text interval range according to a first text length and a second text length, wherein the first text length is a minimum text length in the text length set, and the second text length is a maximum text length in the text length set; Calculate the mean and variance of the text lengths in the text length set, evenly divide the first text interval range into a plurality of first intervals, divide the question-answer pairs in the question-answer pair data set into corresponding first intervals according to the text lengths, and count the number of question-answer pairs in each first interval; Obtaining a first maximum value and a first minimum value of the text interval range of the first target interval, obtaining a first theoretical probability according to the first maximum value, the first minimum value, the mean, the variance and a Gaussian distribution function, and obtaining a first actual sample number ratio according to the number of question-answer pairs in the first target interval; The number of question-answer pairs in the first target interval is adjusted according to the first theoretical probability and the first actual sample number ratio, and a total sample data set is formed based on all the adjusted question-answer pairs in the first target interval.

2. The method for constructing a large model multi-round dialogue according to claim 1, characterized in that: The step of adjusting the number of question-answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio includes: Compare the first theoretical probability with the first actual sample number ratio; When the first theoretical probability is greater than the first actual sample number ratio, a second preset number is calculated, and the second preset number of samples are randomly selected from the first target interval for copying and the first theoretical probability and the first actual sample number ratio are updated, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than a first threshold value; When the first theoretical probability is less than the first actual sample number ratio, calculate a third preset number, randomly take out the third preset number of samples from the first target interval, and update the first theoretical probability and the first actual sample number ratio, so that the difference between the updated first theoretical probability and the updated first actual sample number ratio is less than a first threshold.

3. The method for constructing a large model multi-round dialogue according to claim 1, characterized in that: The step of adjusting the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set comprises: Counting the text lengths of the question-answer pairs in the sub-sample data set to form a sub-text length set, determining a second text interval range according to the third text length and the fourth text length, and adjusting the second text interval range according to the mean and the variance to obtain a third text interval range, wherein the third text length is the minimum text length in the sub-text length set, and the fourth text length is the maximum text length in the sub-text length set; Dividing the third text interval range into a plurality of second intervals evenly, and counting the number of question-answer pairs in each of the second intervals; Obtaining a second maximum value and a second minimum value of the text interval range of the second target interval, obtaining a second theoretical probability according to the second maximum value, the second minimum value, the mean, the variance and the Gaussian distribution function, and obtaining a second actual sample number ratio according to the number of question-answer pairs in the second target interval, where the second target interval is any second interval among the multiple second intervals; The number of question-answer pairs in the second target interval is adjusted according to the second theoretical probability and the second actual sample number ratio so that the target subsample data set conforms to the spatial distribution of the total sample data set.

4. The method for constructing a large model multi-round dialogue according to claim 1, characterized in that: After forming the first multiple rounds of question-answer pairs, the method further includes: Determining a fourth preset number according to the total number of question-answer pairs in the question-answer pair data set and the first preset number; randomly extracting a total of the fourth preset number of times from the total sample data set, extracting the first preset number of question-answer pairs each time, obtaining the fourth preset number of sub-sample data sets, and adjusting the fourth preset number of target sub-sample data sets according to the fourth preset number of sub-sample data sets, and forming the fourth preset number of the first multiple rounds of question-answer pairs according to the fourth preset number of target sub-sample data sets; A second plurality of rounds of question-answer pairs are formed based on the fourth preset number of the first plurality of rounds of question-answer pairs.

5. The method for constructing a large model multi-round dialogue according to claim 4, characterized in that: Determining a fourth preset number according to the total number of question-answer pairs in the question-answer pair data set and the first preset number includes: Substitute the total number of question-answer pairs in the question-answer pair data set and the first preset number into the target calculation formula to calculate the initial number; wherein the target calculation formula is as follows: ； Among them, Epoch is the initial number, N is the total number of question-answer pairs, K is the first preset number; The initial number is rounded to obtain the fourth preset number.

6. A system for constructing a large-scale multi-round dialogue, characterized in that: It includes total sample module, sub-sample module, adjustment module and construction module, among which: A total sample module is configured to obtain a question-answer pair dataset, and adjust the question-answer pair dataset based on a Gaussian distribution to obtain a total sample dataset; A sub-sample module is configured to determine a first preset number according to a preset batch size, and randomly select the first preset number of question-answer pairs from the total sample data set to form a sub-sample data set; an adjustment module, configured to adjust the sub-sample data set to obtain a target sub-sample data set so that the target sub-sample data set conforms to the spatial distribution of the total sample data set; A construction module is configured to form a first multi-round question-answer pair based on the target sub-sample data set, The step of adjusting the question-answer pair data set based on Gaussian distribution to obtain a total sample data set includes: The question-answer pair data set is divided into multiple first intervals, and a first theoretical probability and a first actual sample number ratio of a first target interval are counted. The number of question-answer pairs in the first target interval is adjusted according to the first theoretical probability and the first actual sample number ratio. A total sample data set is formed according to all the adjusted question-answer pairs in the first target interval, where the first target interval is any first interval among the multiple first intervals. The step of dividing the question-answer pair data set into a plurality of first intervals, counting a first theoretical probability and a first actual sample number ratio of a first target interval, adjusting the number of question-answer pairs in the first target interval according to the first theoretical probability and the first actual sample number ratio, and forming a total sample data set according to all adjusted question-answer pairs in the first target interval includes: Counting the text lengths of the question-answer pairs in the question-answer pair data set to form a text length set, and determining a first text interval range according to a first text length and a second text length, wherein the first text length is a minimum text length in the text length set, and the second text length is a maximum text length in the text length set; Calculate the mean and variance of the text lengths in the text length set, evenly divide the first text interval range into a plurality of first intervals, divide the question-answer pairs in the question-answer pair data set into corresponding first intervals according to the text lengths, and count the number of question-answer pairs in each first interval; Obtaining a first maximum value and a first minimum value of the text interval range of the first target interval, obtaining a first theoretical probability according to the first maximum value, the first minimum value, the mean, the variance and a Gaussian distribution function, and obtaining a first actual sample number ratio according to the number of question-answer pairs in the first target interval; The number of question-answer pairs in the first target interval is adjusted according to the first theoretical probability and the first actual sample number ratio, and a total sample data set is formed based on all the adjusted question-answer pairs in the first target interval.

7. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 5 is performed.

Images