Method, device, equipment and storage medium for identifying characteristic drift amplitude

Abstract

The present application provides a method, apparatus, device and storage medium for identifying the amplitude of feature drift. In the method, the device divides the pre-acquired data samples according to the time series to obtain multiple sub-data samples, trains each sub-data sample according to the extreme gradient boosting model to obtain the importance index of each feature in each time period, obtains the drift amplitude of the feature according to the importance index of each feature in different time periods, and identifies the amplitude of feature drift based on the model. The method is simple, easy to use and has high accuracy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data technology, and in particular to a method, device, equipment and storage medium for identifying a characteristic drift amplitude. Background Art

[0002] Machine learning can be used in a variety of scenarios, such as recommendation systems and search systems on the Internet. Machine learning models are needed in the process of machine learning. In order to train a more accurate model, it is necessary to identify various indicators of the features in the collected data. Feature drift refers to the change of features over time. In the current commonly used technical solutions, the identification method of feature drift is mainly based on distribution detection, and various types of collected features are manually processed. Some indicators are obtained based on experience to achieve drift detection.

[0003] However, due to the complexity of the data scenario, analyzing the data and identifying feature drift through manual experience is not only time-consuming but also has low accuracy. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment and storage medium for identifying feature drift amplitude to solve the problem that, due to the complexity of the data scenario, analyzing and identifying feature drift through data through manual experience is not only time-consuming and difficult, but also has low accuracy.

[0005] The first aspect of the present application provides a method for identifying a characteristic drift amplitude, comprising:

[0006] Divide the pre-acquired data samples according to the time series to obtain multiple sub-data samples;

[0007] Each sub-data sample is trained according to the extreme gradient boosting (xgboost) model to obtain the importance index of each feature in each time period;

[0008] According to the importance index of each feature in different time periods, the drift amplitude of the feature is obtained.

[0009] In a specific implementation, obtaining the drift amplitude of each feature according to the importance index of each feature in different time periods includes:

[0010] Based on the importance index of each feature in different time periods, obtain the deviation of each feature over time;

[0011] The deviations of each feature are sorted to obtain the drift magnitude of each feature.

[0012] In a specific implementation, before dividing the pre-acquired data sample according to the time series to obtain a plurality of sub-data samples, the method further includes:

[0013] The data sample is collected and acquired, wherein the data sample includes at least one feature and time information of each feature.

[0014] In a specific implementation, the pre-acquired data samples are divided according to the time series to obtain a plurality of sub-data samples, including:

[0015] According to the time information of each feature, the data sample is divided into time series according to preset time intervals to obtain the multiple sub-data samples.

[0016] A second aspect of the present application provides a device for identifying a characteristic drift amplitude, comprising:

[0017] A partitioning module is used to partition the pre-acquired data samples according to the time series to obtain multiple sub-data samples;

[0018] A processing module, used to train each sub-data sample according to the extreme gradient boosting model to obtain the importance index of each feature in each time period;

[0019] The acquisition module is used to acquire the drift amplitude of each feature according to the importance index of each feature in different time periods.

[0020] Optionally, the acquisition module is specifically used for:

[0021] Based on the importance index of each feature in different time periods, obtain the deviation of each feature over time;

[0022] The deviations of each feature are sorted to obtain the drift magnitude of each feature.

[0023] Optionally, the device further comprises:

[0024] The acquisition module is used to acquire the data sample, wherein the data sample includes at least one feature and time information of each feature.

[0025] Optionally, the division module is specifically used for:

[0026] According to the time information of each feature, the data sample is divided into time series according to preset time intervals to obtain the multiple sub-data samples.

[0027] The third aspect of the present application provides an electronic device, comprising: a processor, a memory and a computer program; the computer program is stored in the memory, and the processor executes the computer program to implement the method for identifying the characteristic drift amplitude provided in any one of the first aspects.

[0028] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to implement the method for identifying the characteristic drift amplitude provided in any one of the first aspects.

[0029] The fifth aspect of the present application provides a computer program product, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method for identifying the characteristic drift amplitude described in the first aspect.

[0030] The embodiments of the present application provide a method, apparatus, device and storage medium for identifying the feature drift amplitude. The device divides the pre-acquired data samples according to the time series to obtain multiple sub-data samples, trains each sub-data sample according to the xgboost model to obtain the importance index of each feature in each time period, obtains the drift amplitude of the feature according to the importance index of each feature in different time periods, and identifies the amplitude of the feature drift based on the model. The device is simple and easy to use and has high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 A flowchart of a first embodiment of a method for identifying a characteristic drift amplitude provided by the present application;

[0033] Figure 2 A flowchart of Embodiment 2 of the method for identifying characteristic drift amplitude provided by the present application;

[0034] Figure 3 A schematic diagram of the structure of a first embodiment of a device for identifying characteristic drift amplitude provided by the present application;

[0035] Figure 4 A schematic diagram of the structure of a second embodiment of a device for identifying characteristic drift amplitude provided by the present application;

[0036] Figure 5 A schematic diagram of the structure of the electronic device entity provided in this application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] Feature drift refers to the change of features over time. Currently, the main solution for detecting feature drift is to manually identify and process various types of collected features, and then detect them in a distributed manner, that is, to obtain some indicators based on experience to achieve drift detection. However, in the process of big data analysis, the data scenarios are complex and the data volume is large. Analyzing and identifying feature drift through manual experience is not only time-consuming but also inaccurate.

[0039] In response to the above problems, the present application provides a method for identifying the drift amplitude of features. The scheme divides the acquired data samples according to time, and trains and identifies them through an extreme gradient boosting model to obtain the drift amplitude of each type of feature. The scheme is described in detail below through several specific embodiments.

[0040] Figure 1 This is a flow chart of a first embodiment of a method for identifying a characteristic drift amplitude provided by the present application, such as Figure 1 As shown, the method for identifying the characteristic drift amplitude can be applied to electronic devices such as servers, cloud servers, and computers that analyze and process data, and can specifically include the following steps:

[0041] S101: Divide the pre-acquired data samples according to the time series to obtain multiple sub-data samples.

[0042] In this step, when analyzing and processing data or conducting model training, you first need to collect a large number of data samples and collect at least one type of feature according to demand. When collecting feature data, you need to obtain the time information corresponding to each feature value, that is, the collected data samples contain multiple types of feature data and the time information corresponding to each feature data.

[0043] After obtaining the data sample, the data sample is divided according to the time information of each feature data, according to a certain time sequence and interval, that is, the data sample is divided into sub-data samples corresponding to multiple time periods according to a certain time series, and each sub-data sample includes multiple types of feature data.

[0044] S102: Train each sub-data sample according to the extreme gradient boosting (eXtreme Gradient Boosting, xgboost) model to obtain the importance index of each feature in each time period.

[0045] Xgboost is one of the boosting algorithms. The idea of ​​the boosting algorithm is to integrate many weak classifiers together to form a strong classifier. Because xgboost is a boosted tree model, it integrates many tree models together to form a strong classifier.

[0046] The idea of ​​the xgboost algorithm is to continuously add trees and perform feature splitting to grow a tree. Each time a tree is added, a new function is actually learned to fit the residual of the previous prediction. When the training is completed and k trees are obtained, to predict the score of a sample, the sample will fall into a corresponding leaf node in each tree according to its characteristics. Each leaf node corresponds to a score. Finally, the scores corresponding to each tree are added up to get the predicted value of the sample.

[0047] In this step, after the electronic device divides the data sample into multiple sub-data samples according to the time series, for each sub-data sample, the xgboost model can be used to train it, and the importance index of each type of feature in the sub-data sample can be obtained. By traversing each sub-data sample, the importance index of each type of feature in each time period can be obtained.

[0048] S103: Obtain the drift amplitude of the feature according to the importance index of each feature in different time periods.

[0049] In this step, based on the importance index of each type of feature in different time periods obtained in the previous step, the deviation of the change of each type of feature as time goes by can be obtained, and then the drift amplitude of each type of feature can be determined.

[0050] In a specific implementation of this step, the electronic device can obtain the deviation of each feature over time according to the importance index of each feature in different time periods, and then sort the deviation of each feature to obtain the drift amplitude of each feature.

[0051] The method for identifying the feature drift amplitude provided in this embodiment is that the electronic device divides the pre-acquired data samples according to the time series to obtain multiple sub-data samples, trains each sub-data sample according to the xgboost model to obtain the importance index of each feature in each time period, obtains the drift amplitude of the feature according to the importance index of each feature in different time periods, and identifies the amplitude of the feature drift based on the model. The method is simple and easy to use and has high accuracy.

[0052] Figure 2 This is a flow chart of Embodiment 2 of the method for identifying characteristic drift amplitude provided by the present application, such as Figure 2 As shown, based on the above embodiment, a specific implementation scheme of the method for identifying the characteristic drift amplitude includes the following steps:

[0053] Before the aforementioned step S101, the solution further includes:

[0054] S104: Collect and obtain data samples, where the data samples include at least one feature and time information of each feature.

[0055] In this step, for the features that need to be analyzed and processed, they are collected in real time according to a certain period. During the collection process, while obtaining each type of feature data, it is necessary to record the time information of each acquisition of the feature data. For example: the user's feature data is collected, and each user's gender, age, income and other features are collected. For example, each time the user's income is collected, the collection time can be recorded, such as June 19, 20**. In more detail, the specific time can be recorded, such as 16:56.

[0056] During the feature collection process, the specific extent to which the time information of the features needs to be recorded can be set according to actual conditions.

[0057] After collecting data samples in the above manner, they are divided according to the time information corresponding to each feature data, as follows.

[0058] Accordingly, in step S101, dividing the pre-acquired data sample according to the time series to obtain a plurality of sub-data samples can be specifically implemented as follows:

[0059] S1011: According to the time information of each feature, the data sample is divided into a time series according to a preset time interval to obtain a plurality of sub-data samples.

[0060] In this step, after obtaining the data sample, the data sample is divided according to the time information of each feature data, according to a certain time sequence and a certain time interval, and divided into sub-data samples corresponding to multiple time periods, each sub-data sample includes multiple types of feature data.

[0061] Specifically, taking the user's characteristic data as an example: all the collected user characteristic data can be divided according to time intervals such as one year, one month or one week. For example, it can be divided into: a sub-data sample corresponding to 2011, a sub-sample set corresponding to 2012... a sub-sample set corresponding to 2019; each sub-sample set includes the user's gender, height, weight, income and other types of characteristics in that year.

[0062] In the specific implementation of this solution, it is recommended to divide the time interval into one evaluation cycle.

[0063] The subsequent steps are consistent with those in the above-mentioned embodiment 1, and are specifically implemented as follows:

[0064] S102: Train each sub-data sample according to the xgboost model to obtain the importance index of each feature in each time period.

[0065] In this step, after the electronic device divides the data sample into multiple sub-data samples according to the time series, the xgboost model can be used to train each sub-data sample to obtain the importance index of each type of feature in the sub-data sample.

[0066] By traversing each sub-data sample, we can obtain the importance index of each type of feature in each time period.

[0067] It can be understood that the higher the value of the importance index of a data feature, the more important the data feature is, and as the importance score decreases, the importance gradually decreases.

[0068] Exemplarily, in addition to using the xgboost model, an importance analysis model trained with the same type of data as the business data of this embodiment can also be used. Therefore, in this embodiment, for each sub-data sample, the acquired feature data can be input into the importance analysis model, and the importance analysis model is used to analyze and process each data feature of the feature data to obtain an importance index.

[0069] The importance index here can be the proportion of each feature, the mean of the feature, the variance value of the feature, or the mean square error of the feature, or other indicators, which are not limited in this solution.

[0070] S103: Obtain the drift amplitude of the feature according to the importance index of each feature in different time periods.

[0071] In this step, based on the importance index of each type of feature in different time periods obtained in the previous step, the deviation of the change of each type of feature as time goes by can be obtained, and then the drift amplitude of each type of feature can be determined.

[0072] Specifically, the electronic device can obtain the deviation of each feature over time according to the importance index of each feature in different time periods, and then sort the deviation of each feature to obtain the drift amplitude of each feature.

[0073] The meaning of this scheme is to obtain the deviation value of the feature over time by comparing the importance ratio of each feature in each time period, and then sort them in chronological order or according to the deviation of the feature type to obtain the drift amplitude of each type of feature.

[0074] The method for identifying the feature drift amplitude provided in the above embodiment is a model-based feature drift method, which can automatically learn the drift amplitude of the feature without manual intervention in the whole process. It identifies the amplitude of the feature drift based on the model, is simple and easy to use, and has high accuracy.

[0075] At the same time, the xgboost model is used to train the sub-data samples to prevent overfitting. The objective function optimization uses the second-order derivative of the loss function with respect to the desired function. At the same time, the xgboost model also supports parallelization. Although the relationship between trees is serial, nodes at the same level can be parallelized. Specifically, for a certain node, the best split point is selected within the node, and the candidate split point gains are calculated in parallel using multiple threads. The training speed is fast.

[0076] In the process of this solution, since the xgboost model adds the processing of sparse data, it can process some relatively sparse feature data more accurately. When the prediction result is already very good, the tree building can be stopped in advance to speed up the training. At the same time, it supports setting sample weights. By adjusting the weights, you can pay more attention to some samples, which can be applied in various technical fields.

[0077] Figure 3 This is a structural diagram of a first embodiment of a characteristic drift amplitude recognition device provided by the present application, such as Figure 3 As shown, the characteristic drift amplitude recognition device 10 includes:

[0078] A division module 11 is used to divide the pre-acquired data sample according to the time series to obtain multiple sub-data samples;

[0079] A processing module 12 is used to train each sub-data sample according to the extreme gradient boosting model to obtain an importance index of each feature in each time period;

[0080] The acquisition module 13 is used to acquire the drift amplitude of each feature according to the importance index of each feature in different time periods.

[0081] The device for identifying the feature drift amplitude provided in this embodiment is used to execute the technical solution provided in any of the aforementioned method embodiments. Its implementation principle and technical effect are similar. The pre-acquired data samples are divided according to the time series to obtain multiple sub-data samples. Each sub-data sample is trained according to the xgboost model to obtain the importance index of each feature in each time period. According to the importance index of each feature in different time periods, the drift amplitude of the feature is obtained, and the amplitude of the feature drift is identified based on the model. It is simple and easy to use and has high accuracy.

[0082] Based on the above embodiment, in a specific implementation manner, the acquisition module 13 is specifically used for:

[0083] Based on the importance index of each feature in different time periods, obtain the deviation of each feature over time;

[0084] The deviations of each feature are sorted to obtain the drift magnitude of each feature.

[0085] Figure 4 This is a structural diagram of Embodiment 2 of the characteristic drift amplitude recognition device provided by the present application, as shown in FIG. Figure 4 As shown, based on the above embodiment, the characteristic drift amplitude recognition device 10 further includes:

[0086] The acquisition module 14 is used to acquire the data sample, where the data sample includes at least one feature and time information of each feature.

[0087] Optionally, the division module 11 is specifically used for:

[0088] According to the time information of each feature, the data sample is divided into time series according to preset time intervals to obtain the multiple sub-data samples.

[0089] The device for identifying characteristic drift amplitude provided in any of the above embodiments is used to execute the technical solution on the electronic device side in the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0090] Figure 5 A schematic diagram of the structure of the electronic device entity provided in this application, such as Figure 5The electronic device 20 shown includes: a processor 21, a memory 22 and a computer program; the computer program is stored in the memory 22, and the processor 21 executes the computer program to implement the technical solution of the method for identifying the characteristic drift amplitude on the electronic device side in any of the aforementioned method embodiments.

[0091] Optionally, the memory 22 may be independent or integrated with the processor 21 .

[0092] When the memory 22 is a device independent of the processor 21, the electronic device may further include:

[0093] The bus 23 is used to connect the processor 21 and the memory 22 .

[0094] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to implement the technical solution of the method for identifying the characteristic drift amplitude on the electronic device side in any of the aforementioned method embodiments.

[0095] The present application also provides a computer program product, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes a solution of any of the aforementioned characteristic drift amplitude identification method embodiments.

[0096] In the specific implementation of the above electronic device, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0097] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: read-only memory (English: read-only memory, abbreviated: ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying a characteristic drift amplitude, characterized in that: include: Collecting and acquiring a data sample, wherein the data sample includes at least one feature and time information of each feature; Divide the pre-acquired data sample according to the time series to obtain multiple sub-data samples; wherein each sub-data sample includes multiple types of feature data; Each sub-data sample is trained according to the extreme gradient boosting model to obtain the importance index of each feature in each time period; According to the importance index of each feature in different time periods, the drift amplitude of the feature is obtained; The data samples acquired in advance are divided according to the time series to obtain multiple sub-data samples, including: According to the time information of each feature, the data sample is divided according to the time series at a preset time interval to obtain the plurality of sub-data samples; the preset time interval is an evaluation period; The extreme gradient boosting model is used to train each sub-data sample to obtain the importance index of each feature in each time period, including: Using the extreme gradient boosting model to train each sub-data sample to obtain the importance index of each feature in each sub-data sample; traversing each sub-data sample to obtain the importance index of each feature in each time period; The step of obtaining the drift amplitude of each feature according to the importance index of each feature in different time periods includes: According to the importance index of each feature in different time periods, the deviation of each feature over time is obtained; the deviation of each feature is sorted to obtain the drift amplitude of each feature.

2. A device for identifying characteristic drift amplitude, characterized in that: include: A collection module, used to collect and obtain data samples, wherein the data samples include at least one feature and time information of each feature; A partitioning module is used to partition the pre-acquired data sample according to the time series to obtain a plurality of sub-data samples; wherein each sub-data sample includes a plurality of types of feature data; A processing module, used to train each sub-data sample according to the extreme gradient boosting model to obtain the importance index of each feature in each time period; An acquisition module, used for acquiring the drift amplitude of each feature according to the importance index of each feature in different time periods; The division module is specifically used for: According to the time information of each feature, the data sample is divided according to the time series at a preset time interval to obtain the plurality of sub-data samples; the preset time interval is an evaluation period; The processing module is specifically used for: Using the extreme gradient boosting model to train each sub-data sample to obtain the importance index of each feature in each sub-data sample; traversing each sub-data sample to obtain the importance index of each feature in each time period; The acquisition module is specifically used for: According to the importance index of each feature in different time periods, the deviation of each feature over time is obtained; the deviation of each feature is sorted to obtain the drift amplitude of each feature.

3. An electronic device, characterized in that: include: A processor, a memory and a computer program; the computer program is stored in the memory, and the processor executes the computer program to implement the method for identifying the characteristic drift amplitude described in claim 1.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the method for identifying the characteristic drift amplitude according to claim 1.

5. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, it is used to implement the method for identifying the characteristic drift amplitude as described in claim 1 above.

Images