Training data generation method and device, computer device, and storage medium

Abstract

The application relates to a training data generation method and device, computer equipment and a storage medium. The application relates to the technical field of information security and artificial intelligence. The method comprises the following steps: acquiring an initial business data set; performing supplementary processing on each initial business data according to a preset supplementary strategy, and performing data cleaning processing on the initial business data after the supplementary processing to obtain each business data; performing clustering processing on each business data to obtain each business data group, and screening, from each business data group, a business data group meeting a preset clustering condition as an initial target training group; extracting a plurality of business data from each initial target training group as initial target training data, and performing data splitting and reorganization processing on each initial target training data to obtain each target training data, wherein the target training data is used for training an artificial intelligence model. The method can reduce attack data in generated business data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, in particular to a training data generation method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] With the development of the financial industry, the financial industry needs various artificial intelligence models to assist in data processing. However, in the process of training the artificial intelligence model, it is often attacked by abnormal client attack data, which causes the artificial intelligence model to be attacked and unable to operate normally. Therefore, how to defend against attack data is the focus of current research.

[0003] The traditional defense method of attack data is to perform data segmentation and reconstruction on all business data at the data level, thereby reducing the impact of potential attack data on the business data set. However, the above method cannot directly eliminate attack data, and simultaneously performing data segmentation and reconstruction on a large amount of normal data and a large amount of abnormal data may greatly damage the data nature of the original normal data, causing a large amount of original normal data to become abnormal, thereby preventing the artificial intelligence model from being trained. SUMMARY

[0004] Therefore, it is necessary to provide a training data generation method, device, computer equipment, computer readable storage medium and computer program product to solve the above technical problems.

[0005] In a first aspect, the present application provides a training data generation method. The method comprises:

[0006] obtaining an initial business data set; the initial business data set comprises a plurality of initial business data;

[0007] According to a preset supplement strategy, each initial business data is supplemented and processed, and the initial business data after the supplement processing is subjected to data cleaning processing to obtain each business data.

[0008] Each business data is subjected to clustering processing to obtain each business data group, and each business data group that satisfies a preset clustering condition is selected from each business data group as an initial target training group.

[0009] From each initial target training group, a plurality of business data are extracted as initial target training data, and each initial target training data is subjected to data splitting and reconstruction processing to obtain each target training data, and the target training data is used for training an artificial intelligence model.

[0010] Optionally, the initial business data is supplemented according to a preset supplement strategy, and the initial business data after the supplement is subjected to data cleaning processing to obtain the business data, including:

[0011] The data attribute of each initial business data is identified, and the first abnormal initial business data having missing data in each initial business data is identified, and the missing data of each first abnormal initial business data is supplemented according to other data of the same data attribute of each first abnormal initial business data to obtain the first business data;

[0012] The second abnormal data in each first business data is identified through an abnormal data detection strategy, and the second abnormal data is deleted to obtain the second business data;

[0013] The second business data is sorted to obtain a data sequence of the initial business data, and the initial business data is subjected to equal division smoothing processing based on the data sequence to obtain the business data.

[0014] Optionally, the data attribute of the initial business data includes a numerical attribute and a non-numerical attribute; and the missing data of each first abnormal initial business data is supplemented according to other data of the same data attribute of each first abnormal initial business data, including:

[0015] For each first abnormal initial business data, in each initial business data, first initial business data having the same data attribute as the first abnormal initial business data and excluding all first abnormal initial business data is selected, and in each first initial business data, each to-be-supplemented data corresponding to the missing data of the first abnormal initial business data is selected according to the missing data of the first abnormal initial business data;

[0016] In a case where the data attribute of the first abnormal initial business data is a numerical attribute, each to-be-supplemented data is subjected to average processing to obtain supplemented data, and the supplemented data is supplemented to the first abnormal initial business data;

[0017] In a case where the data attribute of the first abnormal initial business data is a non-numerical attribute, to-be-supplemented data having the largest number of the same to-be-supplemented data is taken as supplemented data, and the supplemented data is supplemented to the first abnormal initial business data.

[0018] Optionally, the second abnormal data in each first business data is identified through an abnormal data detection strategy, including:

[0019] A quartile box plot of each first business data is established based on the first business data, and an outlier of each first business data is identified based on the quartile box plot.

[0020] The first service data corresponding to the abnormal value satisfying the preset abnormal condition is taken as second abnormal data.

[0021] Optionally, the second service data are sorted to obtain a data sequence of the initial service data, and the initial service data are evenly smoothed based on the data sequence to obtain the service data, including:

[0022] Based on the features of the second service data, a target feature commonly contained in the second service data is determined, and a feature value of the target feature of the second service data is calculated;

[0023] The second service data are sorted in descending order of the feature values of the second service data to obtain a data sequence of the second service data;

[0024] The data sequence is divided into multiple isobath second service data groups by an isobath binning algorithm, and the isobath second service data groups are smoothed to obtain the service data.

[0025] Optionally, the service data are clustered to obtain service data groups, and service data groups satisfying a preset clustering condition are selected from the service data groups as initial target training groups, including:

[0026] The service data are clustered by a clustering analysis strategy to obtain service data groups;

[0027] Outlier service data groups corresponding to outlier service data in the service data groups are identified, and service data groups other than the outlier service data groups are taken as initial target training groups.

[0028] Optionally, a plurality of service data are extracted from the initial target training groups as initial target training data, including:

[0029] A plurality of service data are extracted from the initial target training groups as initial target training data by a random second sampling method.

[0030] Optionally, a plurality of service data are extracted from the initial target training groups as initial target training data by a random second sampling method, including:

[0031] Based on a preset sampling number, a plurality of service data are extracted from each initial target training group as first service data by a random second sampling method, and the first service data in the first service data are extracted as initial target training data.

[0032] In each of the initial target training groups other than each of the initial target training data, a step of extracting a plurality of business data respectively is performed until the number of all the initial target training data meets the number threshold of the initial target training data, and each of the initial target training data is output.

[0033] Optionally, the data splitting and reorganizing processing on each of the initial target training data to obtain each of the target training data comprises:

[0034] The initial target training data is processed by a data splitting algorithm to obtain each of the split data;

[0035] Each of the split data is reorganized by a data reorganizing algorithm to obtain each of the target training data.

[0036] In a second aspect, the present application further provides a training data generation device. The device comprises:

[0037] An acquisition module is configured to acquire an initial business data set; the initial business data set comprises a plurality of initial business data;

[0038] A processing module is configured to perform a supplementing processing on each of the initial business data according to a preset supplementing strategy, and perform a data cleaning processing on the initial business data after the supplementing processing to obtain each of the business data;

[0039] A screening module is configured to perform a clustering processing on each of the business data to obtain each of the business data groups, and screen a business data group meeting a preset clustering condition from each of the business data groups as an initial target training group;

[0040] A reorganizing module is configured to extract a plurality of business data from each of the initial target training groups as initial target training data, and perform a data splitting and reorganizing processing on each of the initial target training data to obtain each of the target training data; the target training data is used for training an artificial intelligence model.

[0041] Optionally, the processing module is specifically configured to:

[0042] Identify a data attribute of each of the initial business data and a first abnormal initial business data having missing data in each of the initial business data, and supplement the missing data of each of the first abnormal initial business data according to each of the other data having the same data attribute of each of the first abnormal initial business data to obtain each of the first business data;

[0043] Identify a second abnormal data in each of the first business data by an abnormal data detection strategy, and delete the second abnormal data to obtain each of the second business data;

[0044] sequencing each of the second business data to obtain a data sequence of each of the initial business data, and performing equal interval smoothing processing on each of the initial business data based on the data sequence to obtain each business data.

[0045] Optionally, the processing module is specifically configured to:

[0046] For each first abnormal initial business data, among the initial business data, a first initial business data having a same data attribute as the first abnormal initial business data and excluding all first abnormal initial business data is selected, and each to-be-supplemented data corresponding to the missing data of the first abnormal initial business data is selected in each first initial business data according to the missing data;

[0047] In a case where the data attribute of the first abnormal initial business data is a numerical attribute, each of the to-be-supplemented data is averaged to obtain a supplemented data, and the supplemented data is supplemented to the first abnormal initial business data;

[0048] In a case where the data attribute of the first abnormal initial business data is a non-numerical attribute, a to-be-supplemented data having a largest number of same to-be-supplemented data is taken as a supplemented data, and the supplemented data is supplemented to the first abnormal initial business data.

[0049] Optionally, the processing module is specifically configured to:

[0050] Based on each of the first business data, a box plot of quartiles of each of the first business data is established, and an abnormal value of each first business data is identified based on the box plot of quartiles;

[0051] A first business data corresponding to an abnormal value satisfying a preset abnormal condition is taken as second abnormal data.

[0052] Optionally, the processing module is specifically configured to:

[0053] Based on features of each second business data, a target feature commonly contained in each of the second business data is determined, and a feature value of the target feature of each of the second business data is calculated;

[0054] Each of the second business data is sequenced in an order from large to small according to the feature value of each of the second business data to obtain a data sequence of each of the second business data;

[0055] The data sequence is divided into a plurality of isobath second business data groups through an isobath binning algorithm, and each of the isobath second business data groups is subjected to data smoothing processing to obtain each business data.

[0056] Optionally, the screening module is specifically used for:

[0057] The business data are clustered by a clustering analysis strategy to obtain business data groups.

[0058] Outlier business data groups corresponding to outlier business data in each business data group are identified, and each business data group except the outlier business data groups is used as an initial target training group.

[0059] Optionally, the reorganization module is specifically used for:

[0060] A plurality of business data are extracted from each initial target training group as initial target training data by random second sampling.

[0061] Optionally, the reorganization module is specifically used for:

[0062] Based on a preset sampling number, a plurality of business data are extracted from each initial target training group as first business data by random second sampling, and the first business data in the preset sampling number are extracted from the first business data as initial target training data.

[0063] In each initial target training group except the initial target training data, the step of extracting a plurality of business data to obtain first business data is returned until the number of all initial target training data meets a number threshold of initial target training data, and each initial target training data is output.

[0064] Optionally, the reorganization module is specifically used for:

[0065] Each initial target training data is segmented by a data segmentation algorithm to obtain segmented data.

[0066] Each segmented data is reorganized by a data reorganization algorithm to obtain target training data.

[0067] In a third aspect, the present application provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method of any one of the first aspect when executing the computer program.

[0068] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method of any one of the first aspect.

[0069] Fifthly, this application provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0070] The aforementioned training data generation method, apparatus, computer equipment, storage medium, and computer program product involve: acquiring an initial business dataset, which includes multiple initial business data sets; supplementing each initial business data set according to a preset supplementation strategy, and then cleaning the supplemented initial business data sets to obtain individual business data sets; clustering each business data set to obtain individual business data groups, and selecting business data groups that meet preset clustering conditions from each business data group as initial target training groups; extracting multiple business data sets from each initial target training group as initial target training data, and performing data splitting and recombination processing on each initial target training data set to obtain individual target training data sets, wherein the target training data sets are used to train an artificial intelligence model. The initial business data is obtained by first supplementing missing data, deleting abnormal data, and smoothing the data. Then, the business data with low attack data content is extracted from the initial target training data through clustering analysis and random sampling. The target training data is obtained by splitting and reorganizing the business data that does not contain attack data. This reduces the abnormal data in the target training data and ensures that the target training data is sufficient to train artificial intelligence data normally. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a training data generation method in one embodiment;

[0072] Figure 2 This is a flowchart illustrating the steps for determining business data in one embodiment;

[0073] Figure 3 This is a flowchart illustrating an example of training data generation in one embodiment;

[0074] Figure 4 This is a structural block diagram of a training data generation device in one embodiment;

[0075] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0077] The training data generation method provided by the embodiments of the present application can be applied to a terminal, a server, a system comprising a terminal and a server, and is realized through the interaction of the terminal and the server. The terminal can include, but is not limited to, various personal computers, notebook computers, tablet computers, and the like. The terminal obtains each service data by supplementing missing data, deleting abnormal data, and smoothing processing of each initial service data, extracts service data containing less attack data from each initial target training data through clustering analysis strategy and random sampling method, and obtains target training data through data splitting and reorganization processing of service data not containing attack data, thereby reducing abnormal data in the target training data and enabling the number of target training data to normally train artificial intelligence data.

[0078] In one embodiment, as shown in Figure 1 A training data generation method is provided. Taking the case of applying the method to a terminal for example, the method comprises the following steps:

[0079] Step S101, obtaining an initial service data set.

[0080] The initial service data set comprises a plurality of initial service data.

[0081] In the embodiment, the terminal obtains a plurality of initial service data to be trained in response to the training operation of a user, and takes the set of the plurality of initial service data as an initial training set. The initial service data can be obtained in the following ways, but is not limited thereto: the initial service data input by the user, the initial service data received from other terminals, or the initial service data generated by a service data generation program. The initial service data can contain attack data. The initial service data is service data with a data label of an artificial intelligence model to be trained, such as financial service data and statistical service data.

[0082] Step S102, supplementing each initial service data according to a preset supplementing strategy, and performing data cleaning processing on the initial service data after the supplementing processing to obtain each service data.

[0083] In this embodiment, the terminal first performs data supplementing processing on the vacancy of each initial service data by a preset supplementing strategy, then deletes the abnormal data of each initial service data after supplementing processing, and finally performs smoothing processing on each initial service data after the above processing process to obtain each service data. Thus, the abnormal data in each initial service data is eliminated, so that each initial service data can be used for training the model. The data impurity processing includes but is not limited to filling in missing data, eliminating abnormal data, smoothing noise data, and correcting inconsistent data, removing noise, filling in null values, missing values, and processing inconsistent data, and then converting the data into a form suitable for training the machine learning model (i.e., the model) through normalization or standardization processing. The specific data impurity processing process will be described in detail later.

[0084] In step S103, the clustering processing is performed on each service data to obtain each service data group, and the service data group satisfying the preset clustering condition is selected from each service data group as an initial target training group.

[0085] In this embodiment, the terminal performs clustering processing on all service data to obtain a plurality of service data groups by a clustering analysis strategy. The clustering analysis method applied by the clustering analysis strategy can be but is not limited to an unsupervised learning clustering analysis method. The terminal selects the service data group satisfying the preset clustering condition from each service data group as an initial target training group. The preset clustering condition can be but is not limited to a preset number ratio, that is, the ratio of the number of service data in the service data group to the number of all service data. The terminal selects the service data group satisfying the preset number ratio as the initial target data group.

[0086] In step S104, a plurality of service data are extracted from each initial target training group as initial target training data, and each initial target training data is subjected to data splitting and reorganization processing to obtain each target training data, which is used for training the artificial intelligence model.

[0087] In this embodiment, the terminal extracts a plurality of service data from each initial target training group by multiple random sampling methods, and each extracted service data is used as initial target training data. The specific extraction process will be described in detail later. The terminal performs data splitting processing on each initial target training data, and performs data reorganization processing on each data after splitting to obtain each target training data. The artificial intelligence model can be but is not limited to a financial service related artificial intelligence model, such as an intelligent appointment model, an intelligent service processing model, an intelligent data statistical model, etc.

[0088] Based on the above scheme, by first supplementing the missing data of each initial business data, deleting abnormal data, and smoothing processing, each business data is obtained, and then the clustering analysis strategy and random sampling method are used to extract business data containing less attack data from each initial target training data, and the data splitting and reorganization processing is performed on the business data not containing attack data to obtain the target training data, thereby reducing the abnormal data in the target training data, and enabling the number of target training data to be normally trained for artificial intelligence data.

[0089] Optionally, as shown in Figure 2 According to the preset supplement strategy, each initial business data is supplemented and processed, and the initial business data after the supplement processing is subjected to data cleaning processing to obtain each business data, including:

[0090] Step S201, identifying the data attribute of each initial business data and the first abnormal initial business data with missing data in each initial business data, and supplementing the missing data of each first abnormal initial business data according to the other data of the same data attribute of each first abnormal initial business data to obtain each first business data.

[0091] In this embodiment, the terminal identifies the data attribute of each initial business data based on each initial business data, and judges whether there is missing data in each initial business data. The terminal regards the initial business data with missing data as the first abnormal initial business data. The terminal divides each first abnormal initial business data according to the data attribute to obtain each first abnormal initial business data group, and groups the other data of each non-first abnormal initial business data according to the data attribute to obtain each initial business data group. The terminal fills in the missing data of each first abnormal initial business data by the other data of the same data attribute according to the data filling strategy corresponding to each data attribute to obtain each first business data. The data attribute can include numerical attribute and non-numerical attribute. The specific supplement process will be described in detail later.

[0092] Step S202, identifying the second abnormal data in each first business data by an abnormal data detection strategy, and deleting the second abnormal data to obtain each second business data.

[0093] In this embodiment, the terminal filters and identifies each second abnormal data in each first business data by an abnormal data detection strategy, and deletes each second abnormal data from each first business data. The specific process of identifying the second abnormal data will be described in detail later.

[0094] Step S203, sorting each second business data to obtain a data sequence of each initial business data, and performing uniform smoothing processing on each initial business data based on the data sequence to obtain each business data.

[0095] In this embodiment, the terminal determines the characteristic values of the second service data based on the second service data, and sorts the second service data in descending order of the characteristic values of each second service data to obtain a data sequence of the second service data. Then, the terminal divides each second service data into a plurality of data groups based on the data sequence, and performs smoothing processing on the second service data in each data group to obtain the service data. The specific smoothing processing process will be described in detail later.

[0096] Based on the above scheme, the impurity processing is performed on each initial service data to obtain the service data, which provides a data basis for subsequent extraction of attack data from the service data and training of the model based on the service data.

[0097] Optionally, the data attributes of the initial service data include numerical attributes and non-numerical attributes; the missing data of each first abnormal initial service data is supplemented according to the other data of the same data attribute of each first abnormal initial service data, including: for each first abnormal initial service data, selecting, from the initial service data, the first initial service data having the same data attribute as the first abnormal initial service data except for all first abnormal initial service data, and selecting, from each first initial service data, the to-be-supplemented data corresponding to the missing data according to the missing data of the first abnormal initial service data; in the case where the data attribute of the first abnormal initial service data is a numerical attribute, performing average processing on the to-be-supplemented data to obtain the supplemented data, and supplementing the supplemented data to the first abnormal initial service data; in the case where the data attribute of the first abnormal initial service data is a non-numerical attribute, selecting the to-be-supplemented data having the largest number of the same to-be-supplemented data as the supplemented data, and supplementing the supplemented data to the first abnormal initial service data.

[0098] In this embodiment, the terminal selects, for each first abnormal initial business data, non-first abnormal initial business data with the same data attribute from the non-first abnormal initial business data according to the data attribute of each first abnormal initial business data. The terminal selects data corresponding to the position of the missing data of the first abnormal initial business data from the non-first abnormal initial business data with the same data attribute as the position of the missing data, as each to-be-supplemented data. Then, the terminal determines the category of the data attribute of the first abnormal initial business data. In the case where the data attribute of the first abnormal initial business data is a numerical attribute, the terminal performs data average processing on each to-be-supplemented data to obtain a supplemented data. The data average processing is numerical data obtained by averaging numerical values. The terminal supplements the first abnormal initial business data based on the supplemented data. In the case where the data attribute of the first abnormal initial business data is a non-numerical attribute, the terminal queries the to-be-supplemented data with the highest occurrence frequency from each to-be-supplemented data, and takes the to-be-supplemented data with the highest occurrence frequency as the supplemented data (i.e., the to-be-supplemented data with the highest number of identical to-be-supplemented data is taken as the supplemented data). The terminal supplements the first abnormal initial business data based on the supplemented data. Similarly, through the above scheme, the terminal supplements all first abnormal business data based on each supplemented data.

[0099] Based on the above scheme, by supplementing the first abnormal initial business data with the supplemented data in the non-first abnormal initial business data, the number of initial business data is maximally retained while the missing data of the first abnormal initial business data is filled in, ensuring the integrity of the initial business data.

[0100] Optionally, the second abnormal data in each business data is identified through an abnormal data detection strategy, including: establishing a quartile box plot of a first business data set based on each first business data, and identifying an abnormal value of each first business data based on the quartile box plot; and taking the first business data corresponding to the abnormal value satisfying a preset abnormal condition as the second abnormal data.

[0101] In this embodiment, the terminal establishes a quartile box plot of the first business data set based on the first business data set through an abnormal data detection strategy. The terminal identifies an abnormal value of each first business data based on the quartile box plot. The abnormal data detection strategy is the establishment procedure of the quartile box plot. For example, Figure 3As shown in the figure, the quartile box plot can be divided into upper limit, lower limit, upper quartile, mean, median, lower quartile, and the like, wherein the quartile is a value corresponding to a 25% percentile point (Q1); the median is a value corresponding to a 50% percentile point (Q2); the upper quartile is a value corresponding to a 75% percentile point (Q3); the upper limit is Q3+1.5(Q3-Q1); the lower limit is Q1-1.5(Q3-Q1); wherein Q3-Q1 represents the interquartile range. The terminal takes a value corresponding to a data point higher than the upper limit or lower than the lower limit as a discrete point value. The terminal takes an abnormal value of first service data corresponding to the discrete point value as an abnormal value satisfying a preset abnormal condition. Specifically, the terminal presets an abnormal threshold, and takes first service data corresponding to an abnormal value lower than the abnormal threshold as second abnormal data.

[0102] Based on the above scheme, the quartile box plot is used to identify abnormal data, and the accuracy of identifying abnormal data is improved.

[0103] Optionally, the second service data is sorted to obtain a data sequence of the initial service data, and the initial service data is evenly smoothed based on the data sequence to obtain the service data, including: determining a target feature commonly contained by the second service data based on the features of the second service data, and calculating the feature value of the target feature of the second service data; sorting the second service data in descending order of the feature value of the second service data to obtain a data sequence of the second service data; and dividing the data sequence into multiple equal-depth second service data groups by an equal-depth binning algorithm, and performing data smoothing on each equal-depth second service data group to obtain the service data.

[0104] In this embodiment, the terminal selects a feature contained by all second service data as a target feature in each feature of each second service data. The features of the service data can be, but are not limited to, data size, data type, data running rate, data sequence number (i.e., the number of important degree sorting times of the data in all data), and the like. The features contained by each service data can be different, and the number of target features selected by the terminal is unique. The terminal calculates the feature value of the target feature of each second service data. The terminal sorts the second service data in descending order of the feature value of the second service data to obtain a data sequence of the second service data. The terminal divides the data sequence into multiple equal-depth second service data groups by an equal-depth binning algorithm. The difference between the first second service data in each equal-depth second service data group and the last second service data in each equal-depth second service data group is the same, but the second service data contained in each equal-depth second service data group is different. The terminal performs data smoothing on each second service data group to obtain the service data.

[0105] For example, the sizes of the second service data are 45, 43, 41, 40, 37, 33, and 32 respectively, and after the data equal division, the sizes of the second service data included in the two groups of equal-depth second service data groups are: the first group: 45, 43, 41, 40, and the second group: 37, 33, 32. The terminal performs data smoothing processing on each equal-depth second service data group according to the mean of the target feature of the equal-depth second service data group, to obtain each smoothed second service data. After the smoothing processing, the feature values of the target features of the second service data are the same, for example, the sizes of the second service data included in the two groups of equal-depth second service data groups after the data equal division are: the first group: 45, 42, 41, 40, and the second group: 37, 33, 32, and after the data smoothing processing, the sizes of the data in each group are: the first group: 42, 42, 42, 42, and the second group: 34, 34, 34.

[0106] Based on the above scheme, by performing smoothing processing on each second service data, the noise and inconsistent data in each second service data are eliminated, and the defense capability against attack data is improved.

[0107] Optionally, the business data is clustered to obtain each business data group, and the business data group satisfying the preset clustering condition is selected from each business data group as an initial target training group, including: performing clustering processing on each business data by a clustering analysis strategy to obtain each business data group; identifying the outlier business data group corresponding to the outlier business data in each business data group, and taking each business data group except the outlier business data group as the initial target training group.

[0108] In this embodiment, the terminal performs clustering processing on each business data by a clustering analysis strategy to obtain each business data group. The terminal presets a business data number threshold in the business data group, and determines whether there is a business data group lower than the business data number threshold in each business data group. In the case that there is a business data group lower than the business data number threshold in each business data group, the terminal takes each business data group lower than the business data number threshold as an outlier business data group. The terminal takes each business data group except the outlier business data group as an initial target training group. The clustering analysis strategy can be, but is not limited to, a clustering strategy corresponding to an unsupervised learning clustering analysis method.

[0109] Based on the above scheme, the outlier business data is taken out after clustering, avoiding the boiling frog attack by injecting only a small amount of toxic data and the label flipping attack by data poisoning. The defense capability against attack data is improved.

[0110] Optionally, the plurality of service data are extracted from each initial target training group as the initial target training data, including: the plurality of service data are extracted from each initial target training group as the initial target training data through random second sampling.

[0111] In this embodiment, the terminal extracts the plurality of service data from each initial target training group through random second sampling, and takes all the extracted service data as the initial target training data. The specific extraction process will be described in detail later.

[0112] Based on the above scheme, the attack data that may exist in the initial target training data can be effectively removed through random second sampling, and the efficiency of training data generation is improved.

[0113] Optionally, the plurality of service data are extracted from each initial target training group as the initial target training data through random second sampling, including: based on a preset sampling number, the plurality of service data are extracted from each initial target training group as the first service data through random second sampling, and the first service data with the preset sampling number are extracted from the first service data as the initial target training data; in each initial target training group except the initial target training data, the step of extracting the plurality of service data to obtain the first service data is returned until the number of all initial target training data meets the number threshold of the initial target training data, and the initial target training data is output.

[0114] In this embodiment, the terminal presets the sampling number and the number threshold, and extracts the plurality of service data from each initial target training group as the first service data through random second sampling. The terminal extracts the first service data with the preset sampling number from the first service data as the initial target training data. The terminal returns to execute the step of extracting the plurality of service data to obtain the first service data in each initial target training group except the initial target training data, and reextracts the service data again until the sum of the number of the initial target training data extracted by the terminal reaches the number threshold, and the terminal stops iteration and outputs the initial target training data.

[0115] Based on the above scheme, the initial target training data in each service data group are extracted through random second sampling, which can prevent the attack data insertion attack of adding a limited number of arbitrary poisoned feature vectors and the data modification attack of modifying the feature vectors or labels of any subset of the service data set, and improves the defense ability against attack data.

[0116] Optionally, each initial target training data is subjected to data splitting and reorganization processing to obtain each target training data, including: each initial target training data is subjected to splitting processing by a data splitting algorithm to obtain each split data; each split data is subjected to two-by-two reorganization processing by a data reorganization algorithm to obtain each target training data.

[0117] In this embodiment, the terminal subjects each initial target training data to splitting processing by a data splitting algorithm to obtain each split data. The data splitting algorithm is any data splitting algorithm that can achieve the above-mentioned manner. The terminal subjects each split data to two-by-two reorganization processing by a data reorganization algorithm to obtain each target training data. The data reorganization algorithm is any data reorganization algorithm that can achieve the above-mentioned manner.

[0118] Based on the above scheme, by subjecting the initial target training data from which a large amount of attack data is removed to data splitting and reorganization processing, the splitting and reorganization of a large amount of normal data and a large amount of attack data is avoided, the usability of the target training data after splitting and reorganization is improved, and the integrity of the target training data is ensured to the greatest extent.

[0119] The present application also provides a training data generation example, as shown in Figure 3 The specific processing process includes the following steps:

[0120] Step S301, obtaining an initial service data set.

[0121] Step S302, for each first abnormal initial service data, selecting, in each initial service data, first initial service data having the same data attribute as the first abnormal initial service data except all first abnormal initial service data, and selecting, in each first initial service data, each to-be-supplemented data corresponding to the missing data according to the missing data of the first abnormal initial service data.

[0122] Step S303, in the case where the data attribute of the first abnormal initial service data is a numerical attribute, performing average processing on each to-be-supplemented data to obtain a supplemented data, and supplementing the supplemented data to the first abnormal initial service data to obtain a first service data.

[0123] Step S304, in the case where the data attribute of the first abnormal initial service data is a non-numerical attribute, taking the to-be-supplemented data with the largest number of the same to-be-supplemented data as a supplemented data, and supplementing the supplemented data to the first abnormal initial service data to obtain a first service data.

[0124] Step S305, based on each first service data, establishing a quartile box plot of the first service data set, and identifying an abnormal value of each first service data based on the quartile box plot.

[0125] Step S306, the first service data corresponding to the abnormal value satisfying the preset abnormal condition is taken as second abnormal data, and the second abnormal data is deleted to obtain each second service data.

[0126] Step S307, based on the data attribute of each second service data, the same target feature between each second service data is determined, and the feature value of the target feature of each second service data is calculated.

[0127] Step S308, according to the order of the feature value of each second service data from large to small, each second service data is sorted to obtain the data sequence of each second service data.

[0128] Step S309, by the equal-depth binning algorithm, the data sequence is divided into multiple equal-depth second service data groups, and each equal-depth second service data group is subjected to data smoothing processing to obtain each service data.

[0129] Step S310, by the clustering analysis strategy, each service data is subjected to clustering processing to obtain each service data group.

[0130] Step S311, the outlier service data group corresponding to the outlier service data in each service data group is identified, and each service data group except each outlier service data group is taken as an initial target training group.

[0131] Step S312, based on the preset sampling number, by the random second sampling method, multiple service data are extracted in each initial target training group to obtain each first service data, and the first service data of the preset sampling number is extracted from each first service data as initial target training data.

[0132] Step S313, in each initial target training group except each initial target training data, the step of extracting multiple service data to obtain each first service data is returned to be executed until the number of all initial target training data satisfies the number threshold of initial target training data, and each initial target training data is output.

[0133] Step S314, by the data segmentation algorithm, each initial target training data is subjected to segmentation processing to obtain each segmentation data.

[0134] Step S315, by the data reorganization algorithm, each segmentation data is subjected to two-by-two reorganization processing to obtain each target training data.

[0135] It should be understood that although each step in the flowchart involved in the above embodiments is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps.

[0136] Based on the same inventive concept, the embodiments of the present application also provide a training data generation device for implementing the above-mentioned training data generation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more training data generation device embodiments provided below can refer to the limitations of the training data generation method in the above text, which will not be repeated here.

[0137] In one embodiment, as shown in Figure 4 a training data generation device is provided, comprising: an acquisition module 410, a processing module 420, a screening module 430 and a reorganization module 440, wherein:

[0138] The acquisition module 410 is configured to acquire an initial service data set; the initial service data set comprises a plurality of initial service data;

[0139] The processing module 420 is configured to perform supplementary processing on each of the initial service data according to a preset supplementary strategy, and perform data cleaning processing on the initial service data after the supplementary processing to obtain each service data.

[0140] The screening module 430 is configured to perform clustering processing on each of the service data to obtain each service data group, and screen a service data group satisfying a preset clustering condition from each of the service data groups as an initial target training group.

[0141] The reorganization module 440 is configured to extract a plurality of service data from each of the initial target training groups as initial target training data, and perform data splitting and reorganization processing on each initial target training data to obtain each target training data, wherein the target training data is used to train an artificial intelligence model.

[0142] Optionally, the processing module 420 is specifically configured to:

[0143] identify data attributes of each initial business data, and first abnormal initial business data in which there is missing data in each of the initial business data, and supplement the missing data of each first abnormal initial business data according to other data of the same data attribute of each first abnormal initial business data, to obtain each first business data;

[0144] identify second abnormal data in each of the first business data through an abnormal data detection strategy, and delete the second abnormal data to obtain each second business data;

[0145] sort each of the second business data to obtain a data sequence of each of the initial business data, and perform equal division smoothing processing on each of the initial business data based on the data sequence to obtain each business data.

[0146] Optionally, the processing module 420 is specifically configured to:

[0147] For each first abnormal initial business data, select, from each of the initial business data, first initial business data other than all first abnormal initial business data and having the same data attribute as the first abnormal initial business data, and select, in each first initial business data, each to-be-supplemented data corresponding to missing data of the first abnormal initial business data according to the missing data;

[0148] In a case where the data attribute of the first abnormal initial business data is a numerical attribute, perform average processing on each of the to-be-supplemented data to obtain supplemented data, and supplement the supplemented data to the first abnormal initial business data;

[0149] In a case where the data attribute of the first abnormal initial business data is a non-numerical attribute, take to-be-supplemented data with the largest number of the same to-be-supplemented data as supplemented data, and supplement the supplemented data to the first abnormal initial business data.

[0150] Optionally, the processing module 420 is specifically configured to:

[0151] establish a quartile box plot of each of the first business data based on each of the first business data, and identify an outlier of each first business data based on the quartile box plot;

[0152] take first business data corresponding to an outlier satisfying a preset abnormal condition as second abnormal data.

[0153] Optionally, the processing module 420 is specifically configured to:

[0154] determine target features commonly contained in the second service data based on features of the second service data, and calculate feature values of the target features of the second service data;

[0155] sort the second service data in descending order of the feature values of the second service data to obtain a data sequence of the second service data;

[0156] divide the data sequence into multiple isobathic second service data groups by an isobathic binning algorithm, and perform data smoothing processing on the isobathic second service data groups to obtain service data.

[0157] Optionally, the screening module 430 is specifically configured to:

[0158] perform clustering processing on the service data by a clustering analysis strategy to obtain service data groups;

[0159] identify outlier service data groups corresponding to outlier service data in the service data groups, and take each service data group except the outlier service data groups as an initial target training group.

[0160] Optionally, the reorganization module 440 is specifically configured to:

[0161] extract multiple service data from each initial target training group as initial target training data by a random second sampling method.

[0162] Optionally, the reorganization module 440 is specifically configured to:

[0163] extract multiple service data from each initial target training group as first service data based on a preset sampling number by a random second sampling method, and extract first service data of the preset sampling number from the first service data as initial target training data;

[0164] return to perform the step of extracting multiple service data from each initial target training group to obtain first service data until the number of all initial target training data meets a number threshold of initial target training data, and output the initial target training data.

[0165] Optionally, the reorganization module 440 is specifically configured to:

[0166] perform segmentation processing on the initial target training data by a data segmentation algorithm to obtain segmentation data;

[0167] perform two-by-two reorganization processing on the segmentation data by a data reorganization algorithm to obtain target training data.

[0168] Each module in the training data generation apparatus described above can be implemented wholly or partially by software, hardware, and combinations thereof. Each module described above can be embedded in a processor in hardware form or independent of the processor in the computer device, or stored in a memory in the computer device in software form, so as to be invoked and executed by the processor to perform the operations corresponding to each module.

[0169] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program is executed by the processor to implement a training data generation method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0170] Those skilled in the art can understand that Figure 5 The structure shown in the above

[0171] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps of the method of any one of the first aspect.

[0172] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the method of any one of the first aspect.

[0173] In one embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps of the method of any one of the first aspect.

[0174] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0175] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to a memory, database or other medium used in the embodiments provided by the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0176] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0177] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for training data generation, characterized in that, The method comprises: obtaining an initial business data set; the initial business data set comprises a plurality of initial business data; According to a preset supplement strategy, each initial business data is supplemented and handled, and the initial business data after the supplement and handling is subjected to data cleaning processing to obtain each business data; Each business data is subjected to clustering processing to obtain each business data group, and a business data group satisfying a preset clustering condition is selected from each business data group as an initial target training group; From each initial target training group, a plurality of business data is extracted as initial target training data, and each initial target training data is subjected to data splitting and reorganization processing to obtain each target training data, which is used to train an artificial intelligence model; According to a preset supplement strategy, each initial business data is supplemented and handled, and the initial business data after the supplement and handling is subjected to data cleaning processing to obtain each business data, comprising: Identify the data attributes of each initial business data and the first abnormal initial business data with missing data in each initial business data, and supplement the missing data of each first abnormal initial business data according to the same data attributes of each other data of each first abnormal initial business data to obtain each first business data; identify the second abnormal data in each first business data through an abnormal data detection strategy, and delete the second abnormal data to obtain each second business data; sort each second business data to obtain a data sequence of each initial business data, and perform equal division smoothing processing on each initial business data based on the data sequence to obtain each business data.

2. The method of claim 1, wherein, The data attributes of the initial business data include numerical attributes and non-numerical attributes; the supplement of the missing data of each first abnormal initial business data according to the same data attributes of each other data of each first abnormal initial business data comprises: For each first abnormal initial business data, select, among all the initial business data, first initial business data with the same data attributes as the first abnormal initial business data except all first abnormal initial business data, and select, in each first initial business data, each to-be-supplemented data corresponding to the missing data according to the missing data of the first abnormal initial business data; In the case where the data attribute of the first abnormal initial business data is a numerical attribute, average processing is performed on each to-be-supplemented data to obtain a supplement data, and the supplement data is supplemented to the first abnormal initial business data; In the case where the data attribute of the first abnormal initial business data is a non-numerical attribute, the to-be-supplemented data with the largest number of the same to-be-supplemented data is selected as a supplement data, and the supplement data is supplemented to the first abnormal initial business data.

3. The method of claim 1, wherein, The identification of the second abnormal data in each first business data through an abnormal data detection strategy comprises: Based on each first business data, a quartile box plot of each first business data is established, and an outlier of each first business data is identified based on the quartile box plot; The first business data corresponding to the abnormal value satisfying the preset abnormal condition is taken as second abnormal data.

4. The method of claim 1, wherein, The sorting of each second business data obtains a data sequence of each initial business data, and the equal interval smoothing processing of each initial business data based on the data sequence obtains each business data, including: Based on the characteristics of each second business data, determine the target characteristics commonly contained in each second business data, and calculate the feature values of the target characteristics of each second business data; According to the order of the feature values of each second business data from large to small, sort each second business data to obtain a data sequence of each second business data; Through the isobath box algorithm, the data sequence is divided into multiple isobath second business data groups, and each isobath second business data group is subjected to data smoothing processing to obtain each business data.

5. The method of claim 1, wherein, The clustering processing of each business data obtains each business data group, and the business data group satisfying the preset clustering condition is selected from each business data group as an initial target training group, including: Through the clustering analysis strategy, each business data is subjected to clustering processing to obtain each business data group; Identify the outlier business data group corresponding to the outlier business data in each business data group, and take each business data group except each outlier business data group as an initial target training group.

6. The method of claim 1, wherein, The multiple business data are extracted from each initial target training group as initial target training data, including: Through random second sampling, multiple business data are extracted from each initial target training group as initial target training data.

7. The method of claim 6, wherein, The multiple business data are extracted from each initial target training group as initial target training data through random second sampling, including: Based on the preset sampling number, through random second sampling, multiple business data are extracted from each initial target training group to obtain each first business data, and the first business data of the preset sampling number is extracted from each first business data as initial target training data; In each initial target training group except each initial target training data, return to execute the step of extracting multiple business data to obtain each first business data until the number of all initial target training data satisfies the number threshold of initial target training data, and output each initial target training data.

8. The method of claim 1, wherein, The data splitting and reorganization processing of each initial target training data obtains each target training data, including: Through a data segmentation algorithm, each initial target training data is segmented to obtain each segmented data; Through a data reorganization algorithm, each segmented data is reorganized to obtain each target training data.

9. A training data generation apparatus characterized by comprising: The device includes: An acquisition module is configured to acquire an initial business data set, wherein the initial business data set includes a plurality of initial business data; A processing module is configured to perform supplementary processing on each initial business data according to a preset supplementary strategy, and perform data cleaning processing on the initial business data after the supplementary processing to obtain each business data; The screening module is configured to perform clustering processing on the service data to obtain service data groups, and screen service data groups satisfying preset clustering conditions from the service data groups as initial target training groups; The reorganization module is configured to extract a plurality of service data from the initial target training groups as initial target training data, and perform data splitting and reorganization processing on each initial target training data to obtain target training data, which is used to train an artificial intelligence model; The processing module is specifically configured to: identify data attributes of each initial service data and first abnormal initial service data in which there is missing data in the initial service data, supplement the missing data of each first abnormal initial service data according to other data of the same data attribute of each first abnormal initial service data to obtain first service data, identify second abnormal data in the first service data through an abnormal data detection strategy, delete the second abnormal data to obtain second service data, sort the second service data to obtain a data sequence of the initial service data, and perform equal division and smoothing processing on the initial service data based on the data sequence to obtain service data. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 8.

Images