Method, device and computer readable storage medium for evaluating features

Abstract

The application discloses a feature evaluation method and device and a computer readable storage medium, and relates to the field of data processing. The feature evaluation method comprises the following steps: adding a first preset number of samples in historical data to a sample pool; every preset time length, adding a second preset number of samples obtained from an online feature calculation system to the sample pool, and deleting the second preset number of existing samples from the sample pool, wherein the online feature calculation system is used for calculating an online feature according to input samples; inputting the samples in the sample pool into a simulated feature calculation system, wherein the simulated feature calculation system is used for calculating an offline feature according to the input samples; and obtaining a calculation performance index of the simulated feature calculation system as an evaluation index of the offline feature. The application can accurately evaluate the offline feature without affecting the operation of the online system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a feature evaluation method and device and a computer readable storage medium. BACKGROUND

[0002] In application scenarios such as risk control, features are usually used for risk assessment. With changes in factors such as the environment and users, risks also change in real time. In order to improve the accuracy of risk assessment, features are also constantly increasing. In the case of continuous iteration of features, in order to keep the system pressure within a controllable range, the relevant indicators of the features need to be monitored. SUMMARY

[0003] The present application relates to the field of data processing, and in particular to a feature evaluation method and device and a computer readable storage medium.

[0004] One technical problem to be solved by an embodiment of the present application is how to accurately evaluate a feature to be put online without affecting the operation of an online system.

[0005] According to a first aspect of some embodiments of the present application, a feature evaluation method is provided, comprising: adding a first preset number of samples in historical data to a sample pool; every preset time interval, adding a second preset number of samples obtained from an online feature calculation system to the sample pool, and deleting the second preset number of existing samples from the sample pool, wherein the online feature calculation system is used to calculate an online feature according to input samples; inputting the samples in the sample pool to a simulated feature calculation system, wherein the simulated feature calculation system is used to calculate a feature to be put online according to input samples; and obtaining a calculation performance indicator of the simulated feature calculation system as an evaluation indicator of the feature to be put online.

[0006] In some embodiments, adding the first preset number of samples in the historical data to the sample pool comprises: adding the first preset number of largest samples in the historical data to the sample pool.

[0007] In some embodiments, adding the first preset number of samples in the historical data to the sample pool comprises: sequentially obtaining samples from the historical data and adding them to the sample pool until the first preset number of samples are obtained, wherein the size of each obtained sample is greater than the size of the samples already added to the sample pool.

[0008] In some embodiments, the historical data is samples obtained from a data service.

[0009] In some embodiments, the first preset number is the upper limit of the number of samples in the sample pool.

[0010] In some embodiments, adding the second preset number of samples obtained from the online feature computing system to the sample pool every preset time interval comprises: sequentially obtaining samples from the online feature computing system every preset time interval and adding them to the sample pool until the second preset number of samples are obtained, wherein each obtained sample has a size greater than the size of the samples already added to the sample pool.

[0011] In some embodiments, the second preset number of samples obtained from the online feature computing system are samples that can be successfully executed in the online feature computing system.

[0012] In some embodiments, the samples used by the online feature computing system are pushed by a data service.

[0013] In some embodiments, the simulated feature computing system comprises a mirror image of the computing process of the online feature computing system and the computing process for the feature to be online.

[0014] In some embodiments, obtaining the computing performance indicators of the simulated feature computing system comprises: obtaining the computing performance indicators of the simulated feature computing system for at least one of the samples obtained from the historical data, the samples obtained from the online feature computing system, or the largest third preset number of samples in the sample pool, respectively; or obtaining the computing performance indicators of the simulated feature computing system for all samples in the sample pool.

[0015] In some embodiments, the computing performance indicators are average computing time.

[0016] According to a second aspect of some embodiments of the present application, there is provided an evaluation device for a feature, comprising: a first sample obtaining module configured to add a first preset number of samples in historical data to a sample pool; a second sample obtaining module configured to add a second preset number of samples obtained from an online feature computing system to the sample pool every preset time interval and delete the second preset number of existing samples from the sample pool, wherein the online feature computing system is used to compute an online feature according to input samples; a simulated computing module configured to input the samples in the sample pool to a simulated feature computing system, wherein the simulated feature computing system is used to compute a feature to be online according to input samples; and an indicator obtaining module configured to obtain computing performance indicators of the simulated feature computing system as evaluation indicators for the feature to be online.

[0017] According to a third aspect of some embodiments of the present application, there is provided an evaluation device for a feature, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any of the foregoing evaluation methods for a feature based on instructions stored in the memory.

[0018] According to a fourth aspect of some embodiments of the present application, a computer readable storage medium is provided, having stored thereon a computer program, wherein the program, when executed by a processor, implements the evaluation method of any of the preceding features.

[0019] Some embodiments of the above-mentioned application have the following advantages or beneficial effects. Embodiments of the present application obtain samples through two ways, i.e. historical data and online feature computing system, and then collect samples suitable for the feature computing system after obtaining a large number of samples, even all samples. Thus, the online system can run normally, and the features to be online can be accurately evaluated.

[0020] Other features and advantages of the present application will become apparent from the following detailed description of illustrative embodiments thereof, which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0022] Figure 1 A flowchart of a feature evaluation method according to some embodiments of the present application is shown.

[0023] Figure 2 A schematic diagram of a feature evaluation system is shown.

[0024] Figure 3 A structural diagram of a feature evaluation device according to some embodiments of the present application is shown.

[0025] Figure 4 A structural diagram of a feature evaluation device according to some embodiments of the present application is shown.

[0026] Figure 5 A structural diagram of a feature evaluation device according to some embodiments of the present application is shown. DETAILED DESCRIPTION

[0027] Clearly, the described embodiments are only some, but not all, embodiments of the present application. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting. For purposes of simplicity and clarity, the descriptive embodiments herein directly or indirectly attach relative terms to reflect internal structural and technical features of the described embodiments. Such descriptive terminology is used for clarity in providing a general understanding of the described embodiments.

[0028] The relative arrangement of parts and steps, numerical expressions, and numerical values set forth in these embodiments are not intended to limit the scope of the application unless otherwise specifically stated.

[0029] It should be understood, of course, that the dimensions of the various parts illustrated in the various figures are chosen primarily to facilitate understanding of the present application and are not intended to limit the scope of the present application.

[0030] Techniques, methods, and apparatus known to the relevant skilled person can not be discussed in detail, but should be considered part of the specification where appropriate.

[0031] In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0032] It should be noted that like reference numerals and letters refer to like items in the following drawings, and thus, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0033] Figure 1 A flowchart of an evaluation method of features according to some embodiments of the present application is shown. As shown, the evaluation method of features of this embodiment includes steps S102-S108. Figure 1

[0034] In step S102, a first preset number of samples in the historical data are added to the sample pool.

[0035] The samples in the embodiments of the present application can be data samples generated in a data service. The samples can include basic information of the corresponding object, or information generated in the running of the data service. For example, when the samples correspond to a piece of user data, the samples can include not only basic information of the user such as age, occupation, gender, etc., but also information generated when the user uses the data service, such as consumption amount, login time, etc.

[0036] ​It should be noted that in the technical solutions of the present disclosure, the collection, use, storage, sharing and transfer of user personal information involved in the processing comply with the relevant legal regulations, and the user needs to be informed and the consent or authorization of the user needs to be obtained, and when applicable, the user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0037] In some embodiments, the first preset number of largest samples in the historical data are added to the sample pool.

[0038] Thus, the sample data entering the sample pool is the current largest data, so that when the sample pool is used for simulation evaluation of the to-be-online feature, the indicators in high-pressure situations can be tested as much as possible, so that the evaluation result is more reliable.

[0039] In some embodiments, samples are sequentially obtained from the historical data and added to the sample pool until the first preset number of samples are obtained, wherein the size of each obtained sample is greater than the size of the sample already added to the sample pool.

[0040] For example, the first preset number is 1000, then the first, second, third, …, 1000 data are sequentially obtained from the historical data, and the size of the second data is greater than that of the first data, the size of the third data is greater than that of the second data, and so on. Thus, the sample data entering the sample pool is the largest data as much as possible, which can also make the evaluation result more reliable.

[0041] In some embodiments, the historical data is a sample obtained from a data service, for example, a sample that has been successfully executed by an online data service. The sample obtained from the historical data is obtained from a real application environment, and therefore has higher credibility, which helps to accurately evaluate the to-be-online feature.

[0042] In step S104, every preset time length, the second preset number of samples obtained from the online feature calculation system are added to the sample pool, and the second preset number of existing samples in the sample pool are deleted, wherein the online feature calculation system is used to calculate the online feature according to the input sample.

[0043] The online feature calculation system is a system that is running online and calculates the online feature. The feature calculation system obtains the calculation result of each feature by inputting the sample and calculating the sample.

[0044] In some embodiments, the second preset number of samples obtained from the online feature computing system are samples that can be successfully executed in the online feature computing system. By obtaining the successfully executed samples from the online feature computing system, the samples entering the sample pool can be more suitable for the feature computing system, i.e., have higher availability. Thus, the accuracy of sample evaluation is improved.

[0045] In some embodiments, every preset time length, samples are sequentially obtained from the online feature computing system and added to the sample pool until the second preset number of samples are obtained, wherein the size of each obtained sample is greater than the size of the sample already added to the sample pool.

[0046] For example, the first preset number is 3, then the first, second and third data are sequentially obtained from the online feature computing system, and the size of the second data is greater than that of the first data, and the size of the third data is greater than that of the second data. Thus, the sample data entering the sample pool is the largest possible data, and the evaluation result can also be more reliable.

[0047] In some embodiments, the first preset number is the upper limit of the number of samples in the sample pool. For example, after adding the first preset number of samples obtained from the historical data to the sample pool, samples are periodically obtained from the online feature computing system at preset time intervals. For example, after using full collection to reach the maximum amount of collection per day, switch to the equal-length fixed-quantity collection mode, and remove one sample from the full-length sample of the day for each collection.

[0048] Thus, the samples in the historical data can be obtained at one time as full-length samples, and real-time samples in the feature computing system can be periodically obtained. In this way, the samples in the sample pool can reach the number of useable samples as soon as possible while maintaining diversity.

[0049] In step S106, the samples in the sample pool are input into the simulated feature computing system, wherein the simulated feature computing system is used to calculate the to-be-online feature according to the input samples.

[0050] In some embodiments, the simulated feature computing system includes a mirror of the computing process of the online feature computing system and a computing process for the to-be-online feature. Thus, the simulated feature computing system can be matched with the currently online running system and can be suitable for the evaluation of the to-be-online feature, so as to be consistent with the system after the feature is online.

[0051] In step S108, the computing performance index of the simulated feature computing system is obtained as the evaluation index of the to-be-online feature.

[0052] In some embodiments, the computing performance index of the simulated feature computing system is obtained for at least one of the samples obtained from historical data, the samples obtained from the feature computing system online, or the largest third preset number of samples in the sample pool. Thus, more targeted evaluation can be performed for the full amount of data samples, the samples executable in the feature computing system, and the largest samples, respectively.

[0053] In some embodiments, the computing performance index of the simulated feature computing system is obtained for all samples in the sample pool. Thus, comprehensive evaluation can be performed for various samples.

[0054] In some embodiments, the computing performance index is the average computing time. By evaluating the time consumption for computing the feature, the system pressure when the feature goes online can be effectively evaluated. Other indexes, such as the consumption of computing resources, can also be used by those skilled in the art as needed, which are not described herein.

[0055] The above embodiments obtain samples through two approaches, historical data and the feature computing system online, respectively. A large number of samples, or even full amount of samples, can be quickly obtained, and then a large number of samples suitable for the feature computing system can be collected. Thus, the to-be-online feature can be accurately evaluated without affecting the operation of the online system.

[0056] In some embodiments, the samples used by the feature computing system online are pushed by the data service.

[0057] Figure 2 An exemplary schematic diagram of the feature evaluation system is shown. As shown in Figure 2 The data service pushes part of the samples to the feature computing service online. The samples pushed to the feature computing service online can be successfully executed or unsuccessfully executed.

[0058] To evaluate the to-be-online feature, first, the top 1000 samples with the largest data size are obtained from the samples of the data service in the last 7 days and added to the sample pool. Then, every 5 minutes, one successfully executed sample is collected from the feature computing system online and added to the sample pool.

[0059] The mirror computing system includes the simulated feature computing system, and further includes the computing process in the feature computing system online. The samples in the sample pool are input to the mirror computing service, for example, timed input, or input when certain conditions are met, or manually triggered input. Thus, evaluation can be performed multiple times under the condition that the samples in the sample pool are constantly changing.

[0060] During the operation of the mirror computing system, evaluation metrics can be collected and displayed, for example, in the form of tables or line charts. Furthermore, corresponding metrics from the online feature calculation system can be obtained from the sample pool for comparison.

[0061] After obtaining the evaluation metrics for the features to be deployed, it can be determined whether the feature meets preset requirements, for example, by using pre-defined conditions. If it does, the feature can be deployed. For example, the corresponding calculation logic for the feature can be added to an online feature calculation system. If it does not meet the requirements, the calculation logic can be further optimized, or other alternative features can be sought. Thus, the new features deployed will have better usability and will not put excessive pressure on the system, ensuring its normal operation.

[0062] The following is for reference. Figure 3 An embodiment of the evaluation apparatus that describes the features of the present invention is described.

[0063] Figure 3 A schematic diagram of the structure of an evaluation apparatus featuring characteristics according to some embodiments of the present invention is shown. For example... Figure 3 As shown, the evaluation device 30 of this embodiment includes: a first sample acquisition module 310, configured to add a first preset number of samples from historical data to a sample pool; a second sample acquisition module 320, configured to add a second preset number of samples obtained from an online feature calculation system to the sample pool every preset time interval, and delete a second preset number of existing samples from the sample pool, wherein the online feature calculation system is used to calculate online features based on the input samples; a simulation calculation module 330, configured to input samples from the sample pool into a simulated feature calculation system, wherein the simulated feature calculation system is used to calculate features to be launched based on the input samples; and an index acquisition module 340, configured to acquire the computational performance index of the simulated feature calculation system as an evaluation index for the features to be launched.

[0064] In some embodiments, the first sample acquisition module 310 is further configured to add a first preset number of the largest samples from the historical data to the sample pool.

[0065] In some embodiments, the first sample acquisition module 310 is further configured to sequentially acquire samples from historical data and add them to a sample pool until a first preset number of samples have been acquired, wherein the size of each acquired sample is greater than the size of the samples already added to the sample pool.

[0066] In some embodiments, historical data are samples obtained from a data service.

[0067] In some embodiments, the first preset quantity is the upper limit of the number of samples in the sample pool.

[0068] In some embodiments, the second sample obtaining module 320 is further configured to obtain samples from the online feature computing system one by one and add them to the sample pool every preset time length until a second preset number of samples are obtained, wherein the size of each obtained sample is greater than the size of the samples already added to the sample pool.

[0069] In some embodiments, the second preset number of samples obtained from the online feature computing system are samples that can be successfully executed in the online feature computing system.

[0070] In some embodiments, the samples used by the online feature computing system are pushed by a data service.

[0071] In some embodiments, the simulated feature computing system includes a mirror image of the computing process of the online feature computing system and the computing process for the feature to be put online.

[0072] In some embodiments, the index obtaining module 340 is further configured to obtain the computing performance index of the simulated feature computing system for at least one of the samples obtained from the historical data, the samples obtained from the online feature computing system, or the largest third preset number of samples in the sample pool; or obtain the computing performance index of the simulated feature computing system for all samples in the sample pool.

[0073] In some embodiments, the computing performance index is the average computing time.

[0074] Figure 4 A structural schematic diagram of a feature evaluation device according to some embodiments of the present application is shown. As shown in the figure, the feature evaluation device 40 of this embodiment includes a memory 410 and a processor 420 coupled to the memory 410, and the processor 420 is configured to execute the feature evaluation method in any one of the preceding embodiments based on instructions stored in the memory 410. Figure 4

[0075] The memory 410 may, for example, include a system memory, a fixed non-volatile storage medium, etc. The system memory, for example, stores an operating system, an application program, a Boot Loader, and other programs, etc.

[0076] Figure 5 A structural schematic diagram of a feature evaluation device according to some other embodiments of the present application is shown. As shown in the figure, Figure 5 ​As shown, the evaluation device 50 of the embodiment comprises a memory 510 and a processor 520, and can further comprise an input / output interface 530, a network interface 540, a storage interface 550, etc. These interfaces 530, 540, 550 and the memory 510 and the processor 520 can be connected through a bus 560, for example. The input / output interface 530 provides a connection interface for display, mouse, keyboard, touch screen, and other input / output devices. The network interface 540 provides a connection interface for various networking devices. The storage interface 550 provides a connection interface for external storage devices such as SD card and U disk.

[0077] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is characterized in that, when executed by a processor, the program realizes the evaluation method of any one of the features.

[0078] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0079] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0080] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0081] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide operational steps for implementing the functions specified in the flowchart Figure 1 one flow or a plurality of flows and / or the functions specified in the block Figure 1 one flow or a plurality of flows and / or the functions specified in the block

[0082] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the protection scope of the present application.

Claims

1. A method for evaluating a feature, comprising: adding a first preset number of samples in historical data to a sample pool; every preset time interval, adding a second preset number of samples obtained from an online feature calculation system to the sample pool and deleting a second preset number of existing samples from the sample pool, wherein the online feature calculation system is configured to calculate an online feature based on input samples, and the second preset number of samples are samples that can be successfully executed in the online feature calculation system; inputting samples in the sample pool to a simulated feature calculation system, wherein the simulated feature calculation system is configured to calculate an offline feature based on input samples, and the simulated feature calculation system comprises a mirror of a calculation process of the online feature calculation system and a calculation process for the offline feature; and obtaining a calculation performance indicator of the simulated feature calculation system as an evaluation indicator of the offline feature.

2. The evaluation method according to claim 1, wherein The adding of the first preset number of samples in historical data to the sample pool comprises: adding a first preset number of largest samples in historical data to the sample pool.

3. The evaluation method according to claim 1 or 2, wherein, The adding of the first preset number of samples in historical data to the sample pool comprises: sequentially obtaining samples from historical data and adding them to the sample pool until a first preset number of samples are obtained, wherein a size of each obtained sample is greater than a size of a sample already added to the sample pool.

4. The evaluation method according to claim 1 or 2, wherein, The historical data are samples obtained from a data service.

5. The evaluation method according to claim 1 or 2, wherein, The first preset number is an upper limit of a number of samples in the sample pool.

6. The evaluation method according to claim 1, wherein The adding of the second preset number of samples obtained from the online feature calculation system to the sample pool every preset time interval comprises: every preset time interval, sequentially obtaining samples from the online feature calculation system and adding them to the sample pool until a second preset number of samples are obtained, wherein a size of each obtained sample is greater than a size of a sample already added to the sample pool.

7. The evaluation method according to claim 1 or 6, wherein The samples used by the online feature calculation system are pushed by a data service.

8. The evaluation method according to claim 1, wherein, The obtaining of the calculation performance indicator of the simulated feature calculation system comprises: for at least one of samples obtained from historical data, samples obtained from the online feature calculation system, or a third preset number of largest samples in the sample pool, obtaining the calculation performance indicator of the simulated feature calculation system; or for all samples in the sample pool, obtaining the calculation performance indicator of the simulated feature calculation system.

9. The evaluation method according to claim 1 or 8, wherein, The calculation performance indicator is an average calculation time.

10. An apparatus for evaluating a feature, comprising: a first sample obtaining module configured to add a first preset number of samples in historical data to a sample pool; a second sample obtaining module configured to, every preset time interval, add a second preset number of samples obtained from an online feature calculation system to the sample pool and delete a second preset number of existing samples from the sample pool, wherein the online feature calculation system is configured to calculate an online feature based on input samples, and the second preset number of samples are samples that can be successfully executed in the online feature calculation system; and a simulated feature calculation system configured to calculate an offline feature based on input samples, and the simulated feature calculation system comprises a mirror of a calculation process of the online feature calculation system and a calculation process for the offline feature. analog computation module, configured to input the sample in the sample pool into an analog feature computation system, wherein the analog feature computation system is used to compute the online feature to be put online according to the input sample, and the analog feature computation system comprises a mirror image of a computation flow of the feature computation system on the line and a computation flow for the online feature to be put online; and an index acquisition module, configured to acquire a computation performance index of the analog feature computation system as an evaluation index of the online feature to be put online. 11.A feature evaluation device, comprising: a memory; and a processor coupled to the memory, configured to execute the feature evaluation method according to any one of claims 1-9 based on instructions stored in the memory. 12.A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the feature evaluation method according to any one of claims 1-9.

Images