Data set construction method and device, equipment, readable storage medium and program product

Abstract

The invention relates to a data set construction method and device, equipment, a readable storage medium and a program product. The method comprises the following steps: acquiring an original data set covering different language structures and knowledge ranges; the original data set comprises question and answer data for the communication service, and evaluating the original data in the original data set according to a pre-constructed general data evaluation index system, a pre-trained data evaluation index system and an SFT data evaluation index system, and obtaining a general quality score, a pre-training suitability score and an SFT potential score corresponding to each piece of original data, and finally constructing a target data set corresponding to the original data set according to each general quality score, the pre-training suitability score, the SFT potential score and the original data. By adopting the method, the high availability of the constructed data set trained by the execution model is ensured, and the accuracy of the data set is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dataset construction method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Pre-training is a fundamental stage in Large Language Models (LLMs), and the quality of the data directly impacts model performance. In recent years, with the increasing size of models, the selection and management of pre-training data has become particularly important. Quality assessment of pre-training data is crucial for optimizing the pre-training process. DataMan uses data labeling and fine-tuning to perform quality scoring and domain identification, thereby selecting high-quality data for training.

[0003] However, current methods of constructing datasets suffer from low accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a dataset construction method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of the constructed dataset in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for constructing a dataset, including:

[0006] Obtain raw datasets covering different language structures and knowledge ranges; the raw datasets include question-and-answer data for communication services;

[0007] According to the pre-constructed general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, the original data in the original dataset are evaluated respectively to obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0008] Based on the general quality scores, pre-training suitability scores, SFT potential scores, and the original data, construct the target dataset corresponding to the original dataset.

[0009] In one embodiment, for any current raw data in the original dataset, the raw data in the original dataset are evaluated according to a pre-constructed general data evaluation index system, a pre-trained data evaluation index system, and an SFT data evaluation index system, respectively, to obtain a general quality score, a pre-training suitability score, and an SFT potential score corresponding to each raw data, including:

[0010] Obtain the sub-general quality scores of the current raw data under the general data evaluation indicators in the general data evaluation indicator system, and obtain the general quality score corresponding to the current raw data based on each sub-general quality score;

[0011] Obtain the sub-pre-training suitability scores of the current raw data under the pre-training data evaluation index in the pre-training data evaluation index system, and obtain the pre-training suitability score corresponding to the current raw data based on each sub-pre-training suitability score;

[0012] Obtain the sub-SFT potential scores of the current raw data under the SFT data evaluation index in the SFT data evaluation index system, and obtain the SFT potential score corresponding to the current raw data based on each sub-SFT potential score.

[0013] In one embodiment, common data evaluation metrics include data volume, text length distribution, character repetition rate, word repetition rate, proportion of special characters, confusion level, knowledge density, and language probability.

[0014] The general quality score corresponding to the current raw data is obtained based on each sub-general quality score, including:

[0015] We will determine the weights of the primary indicators for each of the following factors: data volume, text length distribution, character repetition rate, word repetition rate, proportion of special characters, perplexity, knowledge density, and language probability.

[0016] The general quality scores of each sub-general quality score are weighted and summed based on the weights of each primary indicator to obtain the general quality score corresponding to the current original data.

[0017] In one exemplary embodiment, pre-training data evaluation metrics include DC-PDD, error L2 norm, pre-training-memoization, and pre-training-MIA;

[0018] The pre-training suitability score corresponding to the current raw data is obtained based on the pre-training suitability score of each sub-pre-training suitability score, including:

[0019] Obtain the weights of the second metric for DC-PDD, error L2 norm, pre-training-memoization, and pre-training-MIA respectively;

[0020] The pre-training suitability scores of each sub-pre-training are weighted and summed according to the weights of each second indicator to obtain the pre-training suitability score corresponding to the current original data.

[0021] In one embodiment, SFT data evaluation metrics include Min-k% prob, INSTRUCTMINING, and Rewardscore;

[0022] The SFT potential score corresponding to the current raw data is obtained based on each sub-SFT potential score, including:

[0023] Obtain the weights of the third metric corresponding to Min-k% prob, INSTRUCTMINING, and Rewardscore;

[0024] The SFT potential scores of each sub-SFT potential score are obtained by weighted summation based on the weights of each third indicator, thus obtaining the SFT potential score corresponding to the current original data.

[0025] In one embodiment, a target dataset corresponding to the original dataset is constructed based on various general quality scores, pre-training suitability scores, SFT potential scores, and the original data, including:

[0026] Based on the general quality scores, the original data, and the pre-set first score threshold, construct the first filtered dataset corresponding to the original dataset;

[0027] Based on each pre-training suitability score, the original data, and the pre-set second score threshold, a second filtered dataset corresponding to the original dataset is constructed.

[0028] Based on each SFT potential score, the original data, and the pre-set third scoring threshold, a third screening dataset corresponding to the original dataset is constructed.

[0029] Based on the first, second, and third filtered datasets, construct the target dataset corresponding to the original dataset.

[0030] In an exemplary embodiment, a first filtered dataset corresponding to the original dataset is constructed based on various general quality scores, the original data, and a pre-set first scoring threshold, including:

[0031] Raw data with a general quality score greater than the first score threshold are identified as first candidate data;

[0032] A first filtering dataset is generated based on each first candidate data, or a first preset number of first data are selected from each first candidate data, and a first filtering dataset is generated based on the first preset number of first data.

[0033] In one embodiment, a second filtered dataset corresponding to the original dataset is constructed based on each pre-training suitability score, the original data, and a pre-set second scoring threshold, including:

[0034] The original data whose pre-training suitability score is greater than the second score threshold are identified as the second candidate data;

[0035] A second filtering dataset is generated based on each second candidate data, or a second preset number of second data are selected from each second candidate data, and a second filtering dataset is generated based on the second preset number of second data.

[0036] In one embodiment, a third filtered dataset corresponding to the original dataset is constructed based on each SFT potential score, the original data, and a pre-set third scoring threshold, including:

[0037] The original data whose SFT potential score is greater than the third score threshold are identified as the third candidate data;

[0038] A third filtering dataset is generated based on each third candidate data, or a third preset number of third data are selected from each third candidate data, and a third filtering dataset is generated based on the third preset number of third data.

[0039] In an exemplary embodiment, a target dataset corresponding to the original dataset is constructed based on the first filtered dataset, the second filtered dataset, and the third filtered dataset, including:

[0040] Obtain the first ratio corresponding to the first filtered dataset, the second ratio corresponding to the second filtered dataset, and the third ratio corresponding to the third filtered dataset;

[0041] Based on the first ratio, the second ratio, and the third ratio, the first filtered dataset, the second filtered dataset, and the third filtered dataset are mixed to obtain the target dataset corresponding to the original dataset.

[0042] In one embodiment, the method further includes:

[0043] The general quality score, pre-training suitability score, and SFT potential score corresponding to each raw data are sent to the user terminal; the user terminal is used to display each general quality score, pre-training suitability score, and SFT potential score through a visual interface.

[0044] Secondly, this application also provides a dataset construction apparatus, comprising:

[0045] The acquisition module is used to acquire raw datasets covering different knowledge structures and knowledge scopes; the raw datasets include question and answer data for communication services.

[0046] The evaluation module is used to evaluate the original data in the original dataset according to the pre-built general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, respectively, and obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0047] The building module is used to construct the target dataset corresponding to the original dataset based on various general quality scores, pre-training suitability scores, SFT potential scores, and the original data.

[0048] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0049] Obtain raw datasets covering different language structures and knowledge ranges; the raw datasets include question-and-answer data for communication services;

[0050] According to the pre-constructed general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, the original data in the original dataset are evaluated respectively to obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0051] Based on the general quality scores, pre-training suitability scores, SFT potential scores, and the original data, construct the target dataset corresponding to the original dataset.

[0052] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0053] Obtain raw datasets covering different language structures and knowledge ranges; the raw datasets include question-and-answer data for communication services;

[0054] According to the pre-constructed general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, the original data in the original dataset are evaluated respectively to obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0055] Based on the general quality scores, pre-training suitability scores, SFT potential scores, and the original data, construct the target dataset corresponding to the original dataset.

[0056] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0057] Obtain raw datasets covering different language structures and knowledge ranges; the raw datasets include question-and-answer data for communication services;

[0058] According to the pre-constructed general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, the original data in the original dataset are evaluated respectively to obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0059] Based on the general quality scores, pre-training suitability scores, SFT potential scores, and the original data, construct the target dataset corresponding to the original dataset.

[0060] The aforementioned dataset construction method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire raw datasets covering different language structures and knowledge ranges. These raw datasets include question-and-answer data related to communication services. The raw data in the raw datasets are evaluated according to pre-built general data evaluation index systems, pre-training data evaluation index systems, and SFT data evaluation index systems, respectively, to obtain general quality scores, pre-training suitability scores, and SFT potential scores for each type of raw data. Finally, based on these general quality scores, pre-training suitability scores, SFT potential scores, and the raw data, a target dataset corresponding to the raw dataset is constructed. By evaluating the data using the pre-built general data evaluation index system, pre-training data evaluation index system, and SFT data evaluation index system, and then performing reverse filtering based on the evaluation results, the resulting dataset is constructed. This ensures the high availability of the constructed dataset for model training and improves the accuracy of the dataset. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a diagram illustrating the application environment of a dataset construction method in one embodiment.

[0063] Figure 2 This is a flowchart illustrating a dataset construction method in one embodiment;

[0064] Figure 3 This is a flowchart illustrating the dataset construction method in another embodiment;

[0065] Figure 4 Here is a structural block diagram of a dataset construction apparatus in one embodiment;

[0066] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] The dataset construction method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 obtains a raw dataset from the terminal, including question-and-answer data from user communication services, covering different language structures and knowledge ranges. It evaluates the raw data in the raw dataset according to a pre-built general data evaluation index system, a pre-training data evaluation index system, and an SFT data evaluation index system, obtaining a general quality score, a pre-training suitability score, and an SFT potential score corresponding to each raw data set. Finally, based on the general quality score, pre-training suitability score, SFT potential score, and the raw data, a target dataset corresponding to the raw dataset is constructed. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0069] In one exemplary embodiment, such as Figure 2 As shown, a dataset construction method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 203. Wherein:

[0070] Step 201: Obtain the original dataset covering different language structures and knowledge ranges; the original dataset includes question-and-answer data for communication services.

[0071] Among them, language structure can be understood as the organizational rules and pattern system within a language, which can include phonological / character structure, lexical morphological structure, syntactic structure, semantic structure, and discourse / pragmatic structure. Knowledge scope can be understood as the boundaries of information and concepts that a system possesses, can understand, and can use. Language structure is one of the carriers and forms of expression of knowledge. Communication services can be understood as services that enable the transmission, exchange, and processing of information (voice, data, text, images, video, etc.) between two or more points through a certain technical medium, which can include services such as changes or modifications to call packages.

[0072] For example, server 104 collects user question-and-answer data for communication services through terminal 102, wherein the collected question-and-answer data covers different language structures and knowledge ranges, and an original dataset is constructed based on the collected question-and-answer data.

[0073] Step 202: Evaluate the original data in the original dataset according to the pre-constructed general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, and obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0074] Among them, the general data evaluation index system can be understood as a standard data evaluation system that is suitable for most data evaluation tasks. The pre-training data evaluation index system can be understood as a comprehensive index, method and standard for systematically evaluating and quantifying the quality, characteristics and potential impact on the model's capabilities of pre-training data (massive, multi-source, heterogeneous text / multimodal data used to train the basic model). The SFT data evaluation index system can be understood as a comprehensive set of standards, methods and indicators for systematically evaluating and optimizing the data used in the instruction fine-tuning stage. Its core objective is to ensure that the data can efficiently, safely and controllably teach the pre-trained large model to follow human instructions, understand task intent and output in the desired style and format.

[0075] Optionally, server 104 evaluates the original data in the original dataset according to a pre-constructed general data evaluation index system, a pre-trained data evaluation index system, and an SFT data evaluation index system, thereby obtaining sub-general quality scores of the original data under the general data evaluation index, and obtaining the general quality score of the original data based on each sub-general quality score; and obtaining sub-pre-training suitability scores of the original data under the pre-training data evaluation index, and obtaining the pre-training suitability score of the original data based on each sub-pre-training suitability score; and obtaining sub-SFT potential scores of the original data under the SFT data evaluation index, and obtaining the SFT potential score of the original data based on each sub-SFT potential score.

[0076] Step 203: Construct the target dataset corresponding to the original dataset based on the general quality scores, pre-training suitability scores, SFT potential scores, and the original data.

[0077] For example, server 104 selects first data from each original data based on the general quality score corresponding to each original data, and constructs a corresponding first selected dataset based on the first data; selects second data from each original data based on the pre-training suitability score corresponding to each original data, and constructs a corresponding second selected dataset based on the second data; selects third data from each original data based on the SFT potential score corresponding to each original data, and constructs a corresponding third selected dataset based on the third data; and mixes the first, second and third selected datasets according to a pre-set ratio to obtain the target dataset corresponding to the original dataset.

[0078] In the aforementioned dataset construction method, a raw dataset covering different language structures and knowledge ranges is obtained. This raw dataset includes question-and-answer data related to communication services. The raw data in the raw dataset is evaluated according to a pre-constructed general data evaluation index system, a pre-training data evaluation index system, and an SFT data evaluation index system. This yields a general quality score, a pre-training suitability score, and an SFT potential score for each raw data set. Finally, based on these general quality scores, pre-training suitability scores, SFT potential scores, and the raw data, a target dataset corresponding to the raw dataset is constructed. By evaluating the data using the pre-constructed general data evaluation index system, the pre-training data evaluation index system, and the SFT data evaluation index system, and then performing reverse filtering based on the evaluation results, the resulting dataset is constructed. This ensures the high availability of the constructed dataset for model training and improves the accuracy of the dataset.

[0079] In one embodiment, for any current raw data in the original dataset, the raw data in the original dataset is evaluated according to a pre-constructed general data evaluation index system, a pre-training data evaluation index system, and an SFT data evaluation index system, respectively, to obtain a general quality score, a pre-training suitability score, and an SFT potential score corresponding to each raw data. This includes: obtaining a sub-general quality score of the current raw data under the general data evaluation index system in the general data evaluation index system, and obtaining a general quality score corresponding to the current raw data based on each sub-general quality score; obtaining a sub-pre-training suitability score of the current raw data under the pre-training data evaluation index system in the pre-training data evaluation index system, and obtaining a pre-training suitability score corresponding to the current raw data based on each sub-pre-training suitability score; and obtaining a sub-SFT potential score of the current raw data under the SFT data evaluation index system in the SFT data evaluation index system, and obtaining an SFT potential score corresponding to the current raw data based on each sub-SFT potential score.

[0080] Among them, the sub-general quality score can be understood as the specific score of the original data under the corresponding evaluation index in the general data evaluation index system. Similarly, the sub-pre-training suitability score can be understood as the specific score of the original data under the corresponding evaluation index in the pre-training data evaluation index system. The sub-SFT potential score can be understood as the specific score of the original data under the corresponding evaluation index in the SFT data evaluation index system.

[0081] Optionally, for any current original data in the original dataset, server 104 calculates the sub-general quality score of the current original data under the general data evaluation index in the general data evaluation index system, and performs a weighted sum of each sub-general quality score according to the index weight corresponding to each general data evaluation index to obtain the general quality score corresponding to the current original data; server 104 calculates the sub-pre-training suitability score of the current original data under the pre-training data evaluation index in the pre-training data evaluation index system, and performs a weighted sum of each sub-pre-training suitability score according to the index weight corresponding to each pre-training data evaluation index to obtain the pre-training suitability score corresponding to the current original data; server 104 calculates the sub-SFT potential score of the current original data under the SFT data evaluation index in the SFT data evaluation index system, and performs a weighted sum of each sub-SFT potential score according to the index weight corresponding to each SFT data evaluation index to obtain the SFT potential score corresponding to the current original data.

[0082] Based on the aforementioned implementation method, the current raw data is evaluated using a general data evaluation index system, a pre-trained data evaluation index system, and an SFT data evaluation index system, respectively, thereby obtaining scores for each raw data under different systems, laying a data foundation for subsequent reverse selection of the dataset.

[0083] In one embodiment, the general data evaluation metrics include data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, confusion level, knowledge density, and language probability. A general quality score corresponding to the current original data is obtained based on each sub-general quality score, including: the weights of the first indicators corresponding to the data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, confusion level, knowledge density, and language probability. The general quality score corresponding to the current original data is obtained by weighted summation of each sub-general quality score based on the weights of each first indicator.

[0084] Among them, perplexity can be understood as a commonly used indicator to measure the predictive ability of a probability model (such as a language model).

[0085] For example, server 104 calculates the sub-general quality scores of the current original data under general data evaluation indicators such as data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, confusion level, knowledge density, and language probability. At the same time, it obtains the first indicator weights corresponding to data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, confusion level, knowledge density, and language probability. Finally, based on the weights of each first indicator, it performs a weighted summation of the corresponding sub-general quality scores to obtain the general quality score corresponding to the current original data.

[0086] According to the above implementation method, by using normalized weights and transparent sub-scores, the driving factors of the total score can be explained, thereby improving the accuracy and reliability of the general quality score corresponding to the current raw data.

[0087] In an exemplary embodiment, the pre-training data evaluation metrics include DC-PDD, error L2 norm, pre-training-memoization, and pre-training-MIA. The pre-training suitability score corresponding to the current original data is obtained based on each sub-pre-training suitability score, including: obtaining the second indicator weights corresponding to DC-PDD, error L2 norm, pre-training-memoization, and pre-training-MIA; and performing a weighted summation of each sub-pre-training suitability score according to the weights of each second indicator to obtain the pre-training suitability score corresponding to the current original data.

[0088] Among them, DC-PDD (Data Centered Pretraining Data Diagnosis) can be understood as an indicator that separates data quality assessment from model performance assessment and directly diagnoses based on the characteristics of the data itself. Pretraining-MIA (Membership Inference Attack) can be understood as an indicator that assesses the risk of privacy leakage in training data.

[0089] Optionally, server 104 calculates the sub-pre-training suitability scores of the current original data under pre-training data evaluation metrics such as DC-PDD, error L2 norm, pre-training-memorization, and pre-training-MIA. At the same time, it obtains the weights of the second metrics corresponding to DC-PDD, error L2 norm, pre-training-memorization, and pre-training-MIA. Finally, based on the weights of each second metric, it performs a weighted summation of the corresponding sub-pre-training suitability scores to obtain the pre-training suitability score corresponding to the current original data.

[0090] Based on the aforementioned implementation method, normalized weights and transparent sub-scores can explain the driving factors of the total score, thereby improving the accuracy and reliability of the pre-training suitability score corresponding to the current original data.

[0091] In one embodiment, the SFT data evaluation metrics include Min-k% prob, INSTRUCTMINING, and Rewardscore; obtaining the SFT potential score corresponding to the current original data based on each sub-SFT potential score includes: obtaining the weights of the third indicators corresponding to Min-k% prob, INSTRUCTMINING, and Rewardscore; and performing a weighted summation of each sub-SFT potential score based on the weights of each third indicator to obtain the SFT potential score corresponding to the current original data.

[0092] Among them, Min-k% prob (minimum k% probability) can be understood as a data filtering metric based on the probability prediction of a language model, INSTRUCTMINING (instruction data mining and quality assessment) can be understood as a metric for evaluating and optimizing the quality of instruction fine-tuning data, and Rewardscore (reward model score) can be understood as a metric for the reward model to score the quality of text.

[0093] For example, server 104 calculates the sub-SFT potential scores of the current original data under SFT data evaluation metrics such as Min-k% prob, INSTRUCTMINING, and Rewardscore, and obtains the weights of the third metrics corresponding to Min-k% prob, INSTRUCTMINING, and Rewardscore respectively. Finally, based on the weights of each third metric, the server performs a weighted summation of the corresponding sub-SFT potential scores to obtain the SFT potential score corresponding to the current original data.

[0094] According to the above implementation method, by using normalized weights and transparent sub-scorings, the driving factors of the total score can be explained, thereby improving the accuracy and reliability of the SFT potential score corresponding to the current raw data.

[0095] In one embodiment, the target dataset corresponding to the original dataset is constructed based on various general quality scores, pre-training suitability scores, SFT potential scores, and the original data. This includes: constructing a first filtered dataset corresponding to the original dataset based on various general quality scores, the original data, and a pre-set first scoring threshold; constructing a second filtered dataset corresponding to the original dataset based on various pre-training suitability scores, the original data, and a pre-set second scoring threshold; constructing a third filtered dataset corresponding to the original dataset based on various SFT potential scores, the original data, and a pre-set third scoring threshold; and constructing the target dataset corresponding to the original dataset based on the first filtered dataset, the second filtered dataset, and the third filtered dataset.

[0096] The first scoring threshold can be understood as the minimum general quality requirement that the data used to construct the target dataset must meet; the second scoring threshold can be understood as the minimum pre-training adaptability requirement that the data used to construct the target dataset must meet; and the third scoring threshold can be understood as the minimum SFT potential requirement that the data used to construct the target dataset must meet.

[0097] Optionally, server 104 filters each original data according to a pre-set first scoring threshold and a general quality score corresponding to each original data, and constructs a first filtered dataset corresponding to the original dataset based on the filtered data; server 104 filters each original data according to a pre-set second scoring threshold and a pre-training suitability score corresponding to each original data, and constructs a second filtered dataset corresponding to the original data based on the filtered data; server 104 filters each original data according to a pre-set third scoring threshold and a SFT potential score corresponding to each original data, and constructs a corresponding third filtered dataset based on the filtered data.

[0098] Based on the aforementioned implementation methods, reverse screening under different systems is performed to form different screening datasets, which lays the data foundation for the subsequent mixed screening dataset to form the final target dataset. Secondly, different screening conditions ensure the independence between the screening datasets and guarantee the multi-dimensionality and accuracy of the target dataset.

[0099] In an exemplary embodiment, a first filtered dataset corresponding to the original dataset is constructed based on each general quality score, the original data, and a preset first scoring threshold. This includes: identifying the original data with a general quality score greater than the first scoring threshold as first candidate data; generating the first filtered dataset based on each first candidate data, or selecting a first preset number of first data from each first candidate data, and generating the first filtered dataset based on the first preset number of first data.

[0100] For example, server 104 performs data filtering on the original data according to the first scoring threshold and the general quality score corresponding to the original data, and determines the original data with the general quality score greater than the first scoring threshold as the first candidate data, and generates a first filtering dataset based on each first candidate data, or randomly selects a first preset number of first data from each first candidate data, and generates a first filtering dataset based on the first preset number of first data.

[0101] According to the above implementation method, by performing preliminary screening of the original data based on the first scoring threshold, and further selecting two strategies to construct the corresponding first screening dataset, the construction speed of the first screening dataset is accelerated, and the general quality of the first screening dataset is guaranteed to be qualified.

[0102] In one embodiment, a second screening dataset corresponding to the original dataset is constructed based on each pre-training suitability score, the original data, and a pre-set second scoring threshold. This includes: identifying the original data with pre-training suitability scores greater than the second scoring threshold as second candidate data; generating a second screening dataset based on each second candidate data, or selecting a second preset number of second data from each second candidate data, and generating a second screening dataset based on the second preset number of second data.

[0103] Optionally, the server 104 performs data filtering on the original data based on the second scoring threshold and the pre-training suitability score corresponding to the original data, and determines the original data with the pre-training suitability score greater than the second scoring threshold as the second candidate data, and generates a second filtering dataset based on each second candidate data, or randomly selects a second preset number of second data from each second candidate data, and generates a second filtering dataset based on the second preset number of second data.

[0104] Based on the aforementioned implementation method, by performing preliminary screening of the original data according to the second scoring threshold, two strategies are further selected to construct the corresponding second screening dataset, which accelerates the construction speed of the second screening dataset and ensures that the pre-training suitability of the second screening dataset is qualified.

[0105] In one embodiment, a third screening dataset corresponding to the original dataset is constructed based on each SFT potential score, the original data, and a pre-set third scoring threshold. This includes: identifying the original data with an SFT potential score greater than the third scoring threshold as third candidate data; generating a third screening dataset based on each third candidate data, or selecting a third preset number of third data from each third candidate data, and generating a third screening dataset based on the third preset number of third data.

[0106] For example, server 104 performs data filtering on the original data based on the third scoring threshold and the SFT potential score corresponding to the original data, and determines the original data with SFT potential scores greater than the third scoring threshold as the third candidate data, and generates a third filtering dataset based on each third candidate data, or randomly selects a third preset number of third data from each third candidate data, and generates a third filtering dataset based on the third preset number of third data.

[0107] According to the above implementation method, by performing preliminary screening of the original data based on the third scoring threshold, two strategies are further selected to construct the corresponding third screening dataset, which accelerates the construction speed of the third screening dataset and ensures that the SFT potential in the third screening dataset is qualified.

[0108] In an exemplary embodiment, constructing a target dataset corresponding to the original dataset based on a first filtered dataset, a second filtered dataset, and a third filtered dataset includes: obtaining a first ratio corresponding to the first filtered dataset, a second ratio corresponding to the second filtered dataset, and a third ratio corresponding to the third filtered dataset; and mixing the first filtered dataset, the second filtered dataset, and the third filtered dataset based on the first ratio, the second ratio, and the third ratio to obtain the target dataset corresponding to the original dataset.

[0109] Optionally, server 104 obtains the first ratio corresponding to the first filtered dataset, the second ratio corresponding to the second dataset, and the third ratio corresponding to the third filtered dataset. Based on the first ratio, the second ratio, and the third ratio, the first filtered dataset, the second filtered dataset, and the third filtered dataset are mixed proportionally to obtain the target dataset corresponding to the original dataset.

[0110] Based on the aforementioned implementation method, by performing proportional data mixing on the selected dataset, the high quality of the constructed target dataset in multi-dimensional evaluation metrics is ensured, and the training effect of model training performed using the target dataset is improved.

[0111] In one embodiment, the method further includes sending the general quality score, pre-training suitability score, and SFT potential score corresponding to each raw data to a user terminal; the user terminal is used to display each general quality score, pre-training suitability score, and SFT potential score through a visual interface.

[0112] For example, server 104 sends the general quality score, pre-training suitability score, and SFT potential score corresponding to each raw data to the user terminal. The user terminal displays the general data evaluation indicators such as data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, perplexity, knowledge density, and language probability through a visualization interface; the pre-training data evaluation indicators such as DC-PDD, error L2 norm, pre-training-memorization, and pre-training-MIA; and the SFT data evaluation indicators such as Min-k% prob, INSTRUCTMINING, and Rewardscore. At the same time, it displays the scores of each raw data under different evaluation indicators.

[0113] According to the above implementation method, by displaying indicators and scores on the user terminal's visual interface, the various indicators and analysis results of data evaluation are presented in an intuitive and clear manner, enabling users to understand the characteristics, trends and differences of the data more quickly and accurately, thereby better evaluating the data quality, value and applicability, gaining a deeper understanding of the data's performance and potential problems, and providing clear directions and basis for data optimization and improvement.

[0114] In one exemplary embodiment, such as Figure 3 As shown, a specific implementation of a dataset construction method is provided, wherein:

[0115] Step 1: Create a dataset. The dataset needs to be large enough to cover different language structures and knowledge, and have good diversity, covering different expressions and knowledge ranges. The data quality should be high, not only having correct expressions and accurate knowledge, but also being as safe as possible to avoid duplication.

[0116] Step 2: Build a general data evaluation index system. This system needs to include evaluation of multiple aspects such as data volume, text length distribution, word / phrase repetition rate, special character ratio, confusion level, knowledge density, and language probability.

[0117] Establish a general data evaluation index system, which needs to include evaluation of multiple aspects such as data volume, text length distribution, word / phrase repetition rate, special character ratio, confusion level, knowledge density, and language probability.

[0118] Step 3: Establish an evaluation system for pre-training data. This system should include evaluation metrics such as DC-PDD, error L2 norm, pre-training: memoization, and pre-training: MIA.

[0119] Step 4: Build an SFT data evaluation system, which needs to include data evaluation metrics such as Min-k% prob, INSTRUCTMINING, and Rewardscore.

[0120] Step 5: Visualize the data evaluation results. Present the various indicators and analysis results of the data evaluation in an intuitive and clear way, so that users can understand the characteristics, trends and differences of the data more quickly and accurately, thereby better evaluating the data quality, value and applicability, gaining a deeper understanding of the data performance and potential problems, and providing clear directions and basis for data optimization and improvement.

[0121] Step 6: Reverse filter the dataset. Filter the dataset based on the scoring results. For example, filter the subset A1 of dataset A with a score > 90, or randomly select 500 data points A1 from dataset A.

[0122] Step 7: Data mixing to obtain a new dataset, for example, a subset A1 of dataset A and a subset B1 of dataset B. Mixing A1 and B1 results in a new dataset.

[0123] Compared with the prior art, this application has the following technical advantages:

[0124] 1) Pre-trained models, by learning general patterns from large amounts of data, possess powerful language understanding capabilities. SFT further optimizes the model, making it perform better on specific tasks. In specific tasks, such as text classification and question answering systems, the performance of the model after SFT is significantly improved;

[0125] 2) Pre-trained models demonstrate strong generalization ability across a variety of tasks, and SFT further enhances this ability, enabling models to maintain high performance on similar data that have not been seen before.

[0126] 3) SFT can make more effective use of limited labeled data to optimize model performance, especially when it is difficult or expensive to obtain a large amount of labeled data.

[0127] 4) The pre-training and SFT data evaluation are presented in a visual and systematic way, which can meet the needs of high-quality data extraction of different sizes and data sources, and thus has a lot of generalization ability.

[0128] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0129] Based on the same inventive concept, this application also provides a dataset construction apparatus for implementing the dataset construction method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more dataset construction apparatus embodiments provided below can be found in the limitations of the dataset construction method described above, and will not be repeated here.

[0130] In one exemplary embodiment, such as Figure 4 As shown, a dataset construction apparatus is provided, including: an acquisition module 401, an evaluation module 402, and a construction module 403, wherein:

[0131] The acquisition module 401 is used to acquire raw datasets covering different knowledge structures and knowledge scopes; the raw datasets include question and answer data for communication services;

[0132] Evaluation module 402 is used to evaluate the original data in the original dataset according to the pre-built general data evaluation index system, pre-training data evaluation index system and SFT data evaluation index system, respectively, and obtain the general quality score, pre-training suitability score and SFT potential score corresponding to each original data.

[0133] Module 403 is used to construct the target dataset corresponding to the original dataset based on the general quality scores, pre-training suitability scores, SFT potential scores, and the original data.

[0134] In one embodiment, for any current original data in the original dataset, the evaluation module 402 further includes: a first evaluation submodule, a second evaluation submodule, and a third evaluation submodule, wherein:

[0135] The first evaluation submodule is used to obtain the sub-general quality scores of the current raw data under the general data evaluation indicators in the general data evaluation indicator system, and to obtain the general quality score corresponding to the current raw data based on each sub-general quality score;

[0136] The second evaluation submodule is used to obtain the sub-pre-training suitability scores of the current raw data under the pre-training data evaluation index in the pre-training data evaluation index system, and to obtain the pre-training suitability score corresponding to the current raw data based on each sub-pre-training suitability score.

[0137] The third evaluation submodule is used to obtain the sub-SFT potential score of the current raw data under the SFT data evaluation index in the SFT data evaluation index system, and to obtain the SFT potential score corresponding to the current raw data based on each sub-SFT potential score.

[0138] In one embodiment, the general data evaluation metrics include data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, confusion level, knowledge density, and language probability. The first evaluation submodule is also used to obtain the weights of the first metrics corresponding to each of the data volume, text length distribution, character repetition rate, word repetition rate, special character proportion, confusion level, knowledge density, and language probability. Based on the weights of each first metric, the general quality scores of each sub-general quality score are weighted and summed to obtain the general quality score corresponding to the current original data.

[0139] In an exemplary embodiment, the pre-training data evaluation metrics include DC-PDD, error L2 norm, pre-training-memoization, and pre-training-MIA; the second evaluation submodule is further used to obtain the weights of the second metrics corresponding to DC-PDD, error L2 norm, pre-training-memoization, and pre-training-MIA respectively; and to perform a weighted summation of each sub-pre-training suitability score according to the weights of each second metric to obtain the pre-training suitability score corresponding to the current original data.

[0140] In one embodiment, the SFT data evaluation metrics include Min-k% prob, INSTRUCTMINING, and Rewardscore; the second evaluation submodule is also used to obtain the weights of the third metrics corresponding to Min-k% prob, INSTRUCTMINING, and Rewardscore; and to perform a weighted summation of each sub-SFT potential score based on the weights of each third metric to obtain the SFT potential score corresponding to the current original data.

[0141] In one embodiment, the construction module 403 further includes: a first filtering submodule, a second filtering submodule, a third filtering submodule, and a construction submodule, wherein:

[0142] The first filtering submodule is used to construct the first filtering dataset corresponding to the original dataset based on the general quality scores, the original data, and the pre-set first score threshold.

[0143] The second screening submodule is used to construct the second screening dataset corresponding to the original dataset based on each pre-trained suitability score, the original data, and the pre-set second score threshold.

[0144] The third screening submodule is used to construct the third screening dataset corresponding to the original dataset based on each SFT potential score, the original data, and the pre-set third score threshold.

[0145] The construction submodule is used to construct the target dataset corresponding to the original dataset based on the first filtered dataset, the second filtered dataset, and the third filtered dataset.

[0146] In an exemplary embodiment, the first screening submodule is further configured to identify the original data with a general quality score greater than a first score threshold as first candidate data; generate a first screening dataset based on each first candidate data, or screen out a first preset number of first data from each first candidate data, and generate a first screening dataset based on the first preset number of first data.

[0147] In one embodiment, the second screening submodule is further configured to identify the original data whose pre-training suitability score is greater than the second score threshold as the second candidate data; generate a second screening dataset based on each second candidate data, or screen out a second preset number of second data from each second candidate data, and generate a second screening dataset based on the second preset number of second data.

[0148] In one embodiment, the third screening submodule is further configured to identify the original data whose SFT potential score is greater than the third score threshold as the third candidate data; generate a third screening dataset based on each third candidate data, or screen out a third preset number of third data from each third candidate data, and generate a third screening dataset based on the third preset number of third data.

[0149] In an exemplary embodiment, the construction submodule is further configured to obtain a first ratio corresponding to the first filtered dataset, a second ratio corresponding to the second filtered dataset, and a third ratio corresponding to the third filtered dataset; and to perform data mixing on the first filtered dataset, the second filtered dataset, and the third filtered dataset according to the first ratio, the second ratio, and the third ratio to obtain the target dataset corresponding to the original dataset.

[0150] In one embodiment, the dataset construction apparatus is further configured to send the general quality score, pre-training suitability score, and SFT potential score corresponding to each raw data to a user terminal; the user terminal is configured to display each general quality score, pre-training suitability score, and SFT potential score through a visual interface.

[0151] Each module in the aforementioned dataset construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0152] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores the original dataset, the general quality score corresponding to the original data, the pre-training suitability score, the SFT potential score, and the target dataset. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a dataset construction method.

[0153] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0154] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the dataset construction method of the above embodiment.

[0155] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the dataset construction method of the above embodiment.

[0156] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the dataset construction method of the above embodiments.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0158] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0159] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0160] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data set construction method characterized by comprising: The method comprises: acquiring a raw data set covering different language structures and knowledge ranges; the raw data set comprises question and answer data for a communication service; according to a pre-constructed general data evaluation index system, a pre-training data evaluation index system and an SFT data evaluation index system, evaluating the raw data in the raw data set respectively to obtain a general quality score, a pre-training suitability score and an SFT potential score corresponding to each raw data; according to the general quality score, the pre-training suitability score, the SFT potential score and the raw data, constructing a target data set corresponding to the raw data set.

2. The method of claim 1, wherein, For any one current raw data in the raw data set, according to the pre-constructed general data evaluation index system, the pre-training data evaluation index system and the SFT data evaluation index system, the raw data in the raw data set is evaluated respectively to obtain a general quality score, a pre-training suitability score and an SFT potential score corresponding to each raw data, comprising: acquiring a sub-general quality score of the current raw data under a general data evaluation index in the general data evaluation index system, and obtaining a general quality score corresponding to the current raw data according to each sub-general quality score; acquiring a sub-pre-training suitability score of the current raw data under a pre-training data evaluation index in the pre-training data evaluation index system, and obtaining a pre-training suitability score corresponding to the current raw data based on each sub-pre-training suitability score; acquiring a sub-SFT potential score of the current raw data under an SFT data evaluation index in the SFT data evaluation index system, and obtaining an SFT potential score corresponding to the current raw data according to each sub-SFT potential score.

3. The method of claim 2, wherein, The general data evaluation index comprises data volume, text length distribution, word repetition rate, word repetition rate, special character proportion, perplexity, knowledge density and language probability; The general quality score corresponding to the current raw data is obtained according to each sub-general quality score, comprising: acquiring a first index weight corresponding to each of the data volume, the text length distribution, the word repetition rate, the word repetition rate, the special character proportion, the perplexity, the knowledge density and the language probability; weighting and summing each sub-general quality score based on each first index weight to obtain a general quality score corresponding to the current raw data.

4. The method of claim 2, wherein, The pre-training data evaluation index comprises DC-PDD, error L2 norm, pre-training-memory, pre-training-MIA; The pre-training suitability score corresponding to the current raw data is obtained based on each sub-pre-training suitability score, comprising: acquiring a second index weight corresponding to each of the DC-PDD, the error L2 norm, the pre-training-memory and the pre-training-MIA; weighting and summing each sub-pre-training suitability score according to each second index weight to obtain a pre-training suitability score corresponding to the current raw data.

5. The method of claim 2, wherein, The SFT data evaluation index comprises Min-k% prob, INSTRUCTMINING and Rewardscore. The SFT potential score corresponding to the current raw data is obtained according to each of the sub-SFT potential scores, and the method comprises the following steps: obtaining third index weights corresponding to the Min-k% prob, INSTRUCTMINING, and Rewardscore respectively; weighting and summing each of the sub-SFT potential scores based on the third index weights to obtain the SFT potential score corresponding to the current raw data.

6. The method of claim 1, wherein, The target data set corresponding to the original data set is constructed according to each of the general quality scores, pre-training suitability scores, SFT potential scores, and raw data, and the method comprises the following steps: According to each of the general quality scores, the raw data, and the pre-set first score threshold, a first screening data set corresponding to the original data set is constructed. According to each of the pre-training suitability scores, the raw data, and the pre-set second score threshold, a second screening data set corresponding to the original data set is constructed. According to each of the SFT potential scores, the raw data, and the pre-set third score threshold, a third screening data set corresponding to the original data set is constructed. According to the first screening data set, the second screening data set, and the third screening data set, a target data set corresponding to the original data set is constructed.

7. The method of claim 6, wherein, The target data set corresponding to the original data set is constructed according to each of the general quality scores, the raw data, and the pre-set first score threshold, and the method comprises the following steps: The raw data with a general quality score greater than the first score threshold is determined as first candidate data. The first screening data set is generated based on each of the first candidate data, or a first preset number of first data is screened from each of the first candidate data, and the first screening data set is generated based on the first preset number of first data.

8. The method of claim 6, wherein, The target data set corresponding to the original data set is constructed according to each of the general quality scores, the raw data, and the pre-set first score threshold, and the method comprises the following steps: The raw data with a general quality score greater than the first score threshold is determined as first candidate data. The first screening data set is generated based on each of the first candidate data, or a first preset number of first data is screened from each of the first candidate data, and the first screening data set is generated based on the first preset number of first data.

9. The method of claim 6, wherein, The target data set corresponding to the original data set is constructed according to each of the general quality scores, the raw data, and the pre-set first score threshold, and the method comprises the following steps: The raw data with a general quality score greater than the first score threshold is determined as first candidate data. The first screening data set is generated based on each of the first candidate data, or a first preset number of first data is screened from each of the first candidate data, and the first screening data set is generated based on the first preset number of first data.

10. The method of claim 6, wherein, The target data set corresponding to the original data set is constructed according to each of the general quality scores, the raw data, and the pre-set first score threshold, and the method comprises the following steps: The raw data with a general quality score greater than the first score threshold is determined as first candidate data. The first screening data set is generated based on each of the first candidate data, or a first preset number of first data is screened from each of the first candidate data, and the first screening data set is generated based on the first preset number of first data. The target data set corresponding to the original data set is constructed according to the first screening data set, the second screening data set, and the third screening data set, and the method comprises the following steps: acquire a first proportion corresponding to the first screening dataset, a second proportion corresponding to the second screening dataset, and a third proportion corresponding to the third screening dataset; perform data mixing on the first screening dataset, the second screening dataset, and the third screening dataset according to the first proportion, the second proportion, and the third proportion, to obtain a target dataset corresponding to the original dataset.

11. The method of claim 1, wherein, The method further comprises: send the universal quality score, the pre-training suitability score, and the SFT potential score corresponding to each original data to a user terminal; the user terminal is configured to display the universal quality score, the pre-training suitability score, and the SFT potential score corresponding to each original data through a visual interface.

12. A data set construction apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire original datasets covering different knowledge structures and knowledge ranges; the original datasets comprise question and answer data for communication services; an evaluation module configured to evaluate original data in the original datasets according to a pre-constructed universal data evaluation index system, a pre-training data evaluation index system, and an SFT data evaluation index system, to obtain a universal quality score, a pre-training suitability score, and an SFT potential score corresponding to each original data; a construction module configured to construct a target dataset corresponding to the original datasets according to the universal quality score, the pre-training suitability score, the SFT potential score, and the original data.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 11.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 11.

Images