Feature quality alarm method and device, computer equipment and program product

Abstract

The invention provides a full-life-cycle real-time characteristic quality warning method. Firstly, a calling log of characteristic behaviors of a model calling characteristic project is obtained, a calling statistical value of a target characteristic is obtained from the calling log, and the calling statistical value describes behaviors of the model calling the target characteristic or characteristics of the target characteristic in a period of time. And finally, judging whether to give an alarm or not according to whether the calling statistical value exceeds or is lower than an alarm threshold. The quality of the features output by the feature engineering can be evaluated by calling the statistical value, so that an alarm is given in time when the feature quality is poor, and online features can be optimized during alarm. The feature quality is automatically monitored, and the quality of online features is ensured.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of feature engineering technology, and in particular, to a feature quality alarm method, apparatus, computer equipment, and program product. Background Art

[0002] With the rapid development of information networks, recommendation / risk control machine learning models based on real-time data have been widely used. For traditional machine learning frameworks, feature engineering determines the quality of input data and affects the output results of machine learning models. Therefore, it is necessary to ensure the quality of feature engineering output features.

[0003] However, the existing technology lacks a method to ensure the feature quality of real-time feature engineering output. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a feature quality alarm method, apparatus, computer device, and program product.

[0005] According to a first aspect of one or more embodiments of this specification, a feature quality alarm method is proposed, including:

[0006] Obtaining a call log, where the call log is a log of a feature behavior of a model calling feature engineering;

[0007] For a target feature, obtaining a call statistic value of the target feature within a preset time period from the call log, wherein the call statistic value is used to indicate a statistic describing a behavior of the model calling the target feature within the preset time period, or a characteristic of the target feature called by the model within the preset time period;

[0008] When the call statistic value exceeds or falls below the alarm threshold corresponding to the target feature, an alarm is issued.

[0009] According to a second aspect of one or more embodiments of this specification, a feature quality alarm device is proposed, including:

[0010] A call log acquisition module is used to acquire a call log, where the call log is a log of a feature behavior of a model calling feature engineering;

[0011] A call statistics acquisition module, used for acquiring, for a target feature, a call statistics value of the target feature within a preset time period from the call log, the call statistics value being used to indicate a statistical value describing a behavior of the model calling the target feature within the preset time period, or a characteristic of the target feature called by the model within the preset time period;

[0012] The alarm module is used to issue an alarm when the call statistic value exceeds or falls below the alarm threshold corresponding to the target feature.

[0013] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the feature quality alarm method as described in the first aspect of the embodiments of this specification is implemented.

[0014] According to a fourth aspect of an embodiment of this specification, a computer device is provided, the computer device comprising:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor implements the feature quality alarm method as described in the first aspect of the embodiment of this specification by running the executable instructions.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program product is provided. When the computer program product is executed by a processor, the feature quality alarm method as described in the first aspect of the embodiments of this specification is implemented.

[0019] This specification proposes a feature quality alarm method. First, obtain the call log of the feature behavior of the model calling feature engineering, and obtain the call statistics of the target feature from the call log. The call statistics describe the behavior of the model calling the target feature, or the characteristics of the target feature over a period of time. Finally, determine whether to issue an alarm based on whether the call statistics exceed or fall below the alarm threshold.

[0020] By calling the statistical value, the quality of the features output by the feature engineering can be evaluated, so that timely alarms can be issued when the feature quality is poor, and then the features launched can be optimized when the alarm is issued. Automatic monitoring of feature quality is achieved to ensure the quality of the features launched.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0023] Figure 1 It is a flowchart of a full-link feature quality assurance method shown in this specification according to an exemplary embodiment.

[0024] Figure 2It is a flow chart of a feature quality alarm method shown in this specification according to an exemplary embodiment.

[0025] Figure 3 The flowchart of a method for calculating a call statistic value is shown in this specification according to an exemplary embodiment.

[0026] Figure 4 It is a schematic diagram of a long and short cycle alarm according to an exemplary embodiment of this specification.

[0027] Figure 5 It is a flow chart of an OWA-AHP method according to an exemplary embodiment of this specification.

[0028] Figure 6 It is a block diagram of a feature quality alarm device shown in this specification according to an exemplary embodiment.

[0029] Figure 7 It is a hardware structure diagram of a computer device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0031] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0032] In more and more Internet scenarios, it is necessary to use recommendation / risk control machine learning models based on real-time data. For example, it is necessary to generate features that can describe user behavior, such as user portrait data, based on the user's past operation behavior. And recommend products and content that users are interested in based on user portraits. Specifically, you can recommend advertisements that users may be interested in on the homepage of the application; recommend products that users may be interested in on the sales page of a certain product category, etc.

[0033] In these scenarios, real-time user data is often needed to train or fine-tune the machine learning model. The trained or fine-tuned machine learning model is then used to input real-time user data into the model to obtain the model's prediction results.

[0034] Feature engineering is a method of generating features that can be used by machine learning models or algorithms based on raw data such as past logs. For example, it can generate the real-time user data mentioned above. For machine learning models, the quality of the features output by feature engineering affects the training and prediction results of machine learning models. The quality of the features output by feature engineering mainly includes two aspects: effectiveness and timeliness. Especially for real-time feature engineering, it has higher requirements for the timeliness of features.

[0035] However, in the related art, after the features are launched, there is a lack of a method to ensure the quality of real-time features (such as timeliness and effectiveness).

[0036] Based on this, this specification proposes a feature quality alarm method. First, obtain the call log of the feature behavior of the model calling feature engineering, and obtain the call statistics of the target feature from the call log. The call statistics describe the behavior of the model calling the target feature, or the characteristics of the target feature over a period of time. Finally, determine whether to issue an alarm based on whether the call statistics exceed or fall below the alarm threshold.

[0037] After a feature is launched, it may have problems with the feature itself (for example, as time goes by, the impact of a feature on model prediction and training becomes smaller) or system errors, which may cause the real-time feature to experience a surge or decrease in data volume. Therefore, by calling statistical values, the quality of the features output by feature engineering can be evaluated, so that timely alarms can be issued when the feature quality is poor, and the launched features can be optimized when the alarm is issued. Automatic monitoring of feature quality is achieved to ensure the quality of the launched features.

[0038] Next, the feature quality alarm method involved in this specification will be described in detail.

[0039] In addition to lacking a method to ensure the quality of real-time features output by feature engineering, the related art also lacks a full-link feature quality assurance solution. Therefore, this specification also provides a full-lifecycle real-time feature quality alarm method.

[0040] Therefore, by analyzing the requirements of feature engineering, it can be concluded that the real-time feature management of the entire life cycle provided in this specification includes the following links: feature computing task development, feature configuration online, and feature operation and maintenance. The following will explain the above three links in detail. The specific steps of the three links are as follows: Figure 1 shown.

[0041] First, the features to be launched can be determined based on the amount of data of each feature, whether the offline and online features are consistent, etc., and the features to be launched can be configured and launched. Specifically, this process corresponds to the steps of computing task development and feature configuration and launch.

[0042] 1. Computing task development

[0043] Computing task development is to determine the features that can be put online. Figure 1 As shown, the process includes the following three steps:

[0044] (1) Data volume assessment and performance assurance

[0045] The performance of feature development after going online can be preliminarily evaluated by obtaining the amount of real-time feature data consumed by the model.

[0046] Specifically, performance assurance needs to be based on data volume assessment. If the data volume of a business scenario is large, it will easily lead to poor computing task performance and high subsequent operation and maintenance costs, which will lead to a low return on investment (ROI).

[0047] Therefore, we can determine whether to launch a feature based on the amount of data and the actual needs of feature development. For example, if the amount of data for a feature is large, the model development cost is limited, and the feature is of low importance, we can choose not to launch the feature. Another example is if the amount of data for a feature is large, but the model development cost can be further improved, and the feature itself is also relatively important, we can choose to launch the feature.

[0048] The amount of feature data can be obtained based on the user's behavior log. The user's behavior log is a log that records the user's interaction with the application or web page. Based on the log, the amount of data of various features of the user can be statistically obtained.

[0049] For example, if a feature is the sequence of user clicks on icons in the application interface, then we can extract the click sequence based on the behavior logs of each user, and count the data volume of the click sequence over a period of time to perform performance analysis.

[0050] (2) Offline and online consistency check

[0051] Obtain the statistical values ​​or feature details of offline features and real-time features in the same time period (hourly or daily granularity) and compare the offline features with the real-time features.

[0052] Specifically, real-time features can refer to features obtained from a period of real-time statistics. Offline features can refer to features obtained from existing logs over a period of time. Due to reasons such as interfaces, the statistical values ​​of offline features and real-time features are generally different.

[0053] If the difference between the two is large, it may prove that there is an error in the real-time feature, affecting the quality of feature engineering. Therefore, it is necessary to check the consistency between offline and online features. Specifically, the statistical values ​​(such as data volume) and details (such as whether the features obtained by a certain user in a specific time period are consistent) of offline features and real-time features in the same time window can be compared. If the error between the two is reasonable, it means that the feature is acceptable and can be put online. If the error between the two is unreasonable, it can be determined whether to put the feature online based on actual needs. For example, the importance of the feature can be used to analyze whether to put the feature online; the reason for the large error between the two can be analyzed to determine whether to put the feature online. For example, if the large error between the two is due to problems with offline feature statistics, the feature can be put online.

[0054] (3) R&D norms and constraints

[0055] R&D specifications can be formulated based on specific business and technical scenarios.

[0056] Specifically, R&D specifications need to be formulated in combination with specific business and technical scenarios (such as agreeing on task naming methods, field output formats and data types, etc.) to avoid affecting other upstream and downstream related tasks due to the launch of the task.

[0057] 2. Feature configuration online

[0058] like Figure 1 As shown in the figure, feature configuration launch includes the following two steps:

[0059] (1) Feature configuration and debugging

[0060] Feature configuration and debugging, that is, the features that need to be launched in step 1 of the launch.

[0061] Specifically, when going online, you can go online according to the requirements of the R&D platform and perform online operations in combination with their respective processes. After the online launch is completed, you can also debug the online real-time features, such as simply running the system to check whether there are any abnormalities in the performance of the real-time features after going online.

[0062] (2) Feature asset synchronization

[0063] After the feature configuration is online, the feature assets can also be synchronized to the data asset database and the feature dictionary can be built in the asset database.

[0064] Specifically, in the related technologies, when performing feature management, data asset management is not combined. Therefore, when there is a problem with feature engineering, it is not possible to quickly troubleshoot the error by combining the relevant upstream and downstream links of the feature engineering problem.

[0065] In the solution of this specification, the feature assets are synchronized to the asset database. The asset database records the real-time features of each online, as well as the upstream and downstream information, links, logs, etc. associated with each real-time feature. When problems with real-time features are found later, you can quickly filter from the asset database according to the screening conditions (such as the feature name of the real-time feature, the status of the real-time feature, the data source, etc.), and filter out the logs associated with the real-time features.

[0066] Compared with the related data, data asset management is not combined with feature management, which leads to the need for manual maintenance of the online features and related information, which is a huge workload and prone to omissions. The method in this manual adds data asset management to the entire link of feature quality assurance, which facilitates timely troubleshooting when problems are found later.

[0067] 3. Feature operation and maintenance.

[0068] Specifically, it is to maintain the real-time features that have been launched to quickly find out whether there are problems with the features so that they can be repaired in time to ensure the quality of the features. Next, a feature quality alarm method provided in this manual will be described in detail. Figure 2 As shown, the method comprises the following steps:

[0069] Step 201, obtain the call log.

[0070] The call log is a log of the feature behavior of the model calling feature engineering.

[0071] The call log records the behavior of the real-time features of the model consumption. In order to analyze the data such as the number of each feature, it is necessary to analyze based on the call behavior of the model, so you need to obtain the call log first.

[0072] Step 203: For the target feature, obtain the call statistics of the target feature within a preset time period from the call log.

[0073] The call statistics are used to indicate the behavior of the model calling the target feature within a preset time period, or the characteristics of the target feature called by the model within a preset time period.

[0074] This step corresponds to Figure 1 Specifically, call statistics that can describe feature quality are obtained by analyzing call logs.

[0075] The target feature may be a real-time feature. In other embodiments, the target feature may also be an offline feature. Although the above analysis is based on real-time features, the method of this specification may also be applied to offline features.

[0076] The target feature can be any one or features that have been launched in feature engineering, and the target feature can also include all the features that have been launched. This specification does not limit the number of target features. For the convenience of description, one target feature is used as an example for description below.

[0077] The specific implementation method of this step may be to obtain the feature call log and its corresponding format, and perform field parsing on different fields according to the format, so as to obtain the feature call information.

[0078] In addition, as described above, a feature dictionary may be preset. In this case, the target feature is the feature included in the feature dictionary. The model may call real-time features or offline features. In the case where the method in this specification only maintains one feature, it is also possible to determine whether the called feature is the target feature based on the feature dictionary for the feature called in the call log.

[0079] In this way, the feature sequences mentioned above after analysis can be traversed to determine whether each feature sequence is in the asset dictionary. Then, only the target features in the feature dictionary can be called for statistical value calculation to improve efficiency.

[0080] After parsing the log, we can further calculate the call statistics of the target feature. Figure 1 The example of calling statistics is used to illustrate the Figure 1 The examples shown do not limit the present specification. Any statistical value that can be used to evaluate the quality of a feature can be used as a call statistical value in the present specification.

[0081] As mentioned above, call statistics can describe the behavior of model call features or describe the characteristics of target features. Here we will explain the two types of call statistics respectively. Figure 1 As shown, the call statistics may include one or more of the following:

[0082] 1) Timeliness

[0083] The call statistics are used to describe the average delay time between the generation and the call of the feature value of the target feature. The generation refers to the generation of feature engineering, and the call refers to the consumption by the model. In other words, timeliness is used to describe the delay time of real-time tasks. The average delay time refers to the average delay time of the target features of multiple objects.

[0084] If the delay time is high, there is a problem with the timeliness of the characterization feature, and targeted improvements can be made.

[0085] 2) Call volume

[0086] The call statistics are used to describe the call volume of the target feature within a preset time period. Since the model generally calls all features generated by feature engineering, the overall data volume of the target feature can be evaluated through the call volume, a model call behavior. If the data volume is low, it may indicate that the feature has little effect on the training of the model. If the data volume is high, there may be a problem of low ROI. It can be seen that the call volume exceeding or falling below the corresponding threshold may indicate a problem with the target feature, and then the feature can be repaired and adjusted in a targeted manner.

[0087] As for the specific method of obtaining the call volume, it can be to obtain the call count of each feature in each call log, and determine whether it is in the real-time feature asset dictionary. If it exists, the call count is accumulated.

[0088] 3) Daily / real-time share

[0089] This call statistic is used to describe the ratio of the target feature's call volume to the total call volume of all features within a preset time period. It is similar to the call volume. The call volume evaluates the call behavior of the target feature from the order of magnitude, while the call statistic evaluates the call behavior of the target feature from the proportion.

[0090] The specific method for obtaining this feature can be: accumulate the number of feature calls in each call log, and calculate the ratio of the target feature call volume to all feature call volumes in a certain time interval. If the interval is extended to the daily interval, it is the daily ratio, and if it is instantaneous, it is the real-time ratio.

[0091] 4) Feature correlation (contribution)

[0092] The call statistics are used to describe the impact of the target feature on the result.

[0093] As for its specific calculation method, the contribution of the target feature to the overall revenue can be calculated based on specific business logic.

[0094] Specifically, you can first obtain the ROI when the model calls all features, and then obtain the ROI when the model calls features other than the target feature, and compare the two ROIs to determine the relevance of the features. The specific form of ROI varies depending on the scenario. For example, for the scenario of recommending ads that users may be interested in, ROI can be determined based on the user's click-through rate and the amount of money spent by the user on the page.

[0095] 5) Feature positive sample rate and null / 0 value ratio

[0096] This call statistic is used to describe the proportion of the first feature value in the target feature to the total number of target features within a preset time period. If the positive sample rate is too low, or the proportion of null values / 0 values ​​is high, it means that the feature has few valid values, which makes the feature have little impact on the model and may have low effectiveness.

[0097] Among them, the feature positive sample rate and the proportion of null values / 0 values ​​are proposed for different scenarios.

[0098] Specifically, when the target feature is a binary feature, the first feature value is any one of the feature values. When the binary values ​​are 0 and 1 respectively, the above binary feature can also be called a 0 / 1 type feature. When 1 is a positive sample and 0 is a negative sample, when the first feature value is 1, the feature positive sample rate is calculated.

[0099] As for the specific meaning of binary features, in the advertising push scenario, a binary feature may be whether the user clicks a specific button in the application page. If the user clicks the button, the feature value may be 1. Then, the user's preference may be predicted based on whether the user clicks the button.

[0100] When the target feature is a sequence feature, the first feature value is a null value or a value of 0. Specifically, for the target feature, the ratio of null values / 0 values ​​to the total feature length in all target features called within a preset time period is calculated.

[0101] A sequential feature may be, for example, the order in which a user clicks buttons in a specific interface other than the home page of an application.

[0102] In addition, when the target feature is a sequence feature, the call statistics may further include the following 6 and 7.

[0103] 6) Feature coverage

[0104] The feature coverage rate is used to indicate the proportion of the number of objects that include the target feature to the total number of objects within a preset time period. Specifically, the above-mentioned objects may be users, that is, the proportion of the number of people covered by the feature to the total number of people entering the scene. Specifically, the number of people covered by the feature can be determined based on the feature call volume obtained from the call log. The total number of people entering the scene may be the total number of people who visit a specific page, which can be calculated based on the page views, and the page views can be determined based on the user behavior log. The page in the page views has different meanings in different scenarios. For example, in a scenario where insurance is recommended to users, the page may be the insurance sales page entered by the user.

[0105] When the feature coverage is low, there may be a problem that the feature has little impact on the model. Therefore, the target feature can be processed adaptively.

[0106] 7) Label distribution uniformity

[0107] Label distribution uniformity is also referred to as feature coverage uniformity in this specification. Feature coverage uniformity is used to indicate the uniformity of feature coverage rates corresponding to different target features.

[0108] It should be noted that the label here does not refer to the features of the training samples, but to the coverage of the features for different groups of people. Similar to feature coverage, feature coverage is used to evaluate the number of people covered by a certain feature, while feature coverage uniformity is used to describe whether the number of people covered by different features is uniform.

[0109] If multiple target features are unevenly distributed, it means that some features may have a small impact on the model and low effectiveness.

[0110] Step 205 is explained with reference to a specific example, which does not limit the present specification. Next, we will take the example of recommending insurance products that the user may be interested in in the insurance sales scenario, and use the target features of whether the user has searched on the homepage of the insurance sales interface (0 / 1 type feature) and the click sequence of the recommended insurance products on the homepage of the insurance sales interface (sequence type feature) as examples for explanation. The specific implementation of the above process can be found in Figure 3 Next, we will combine Figure 3 Provide further explanation.

[0111] You can develop user-defined functions (UDF) and input parameters in advance based on the specific data volume and business indicator requirements. UDF records how to calculate the statistical values ​​of the parsed features.

[0112] First, read the log, split the key-value pairs by the delimiter, and get the key-value pair list of each real-time target feature. Each key-value pair records the name of a feature called by the model (key) and the specific feature value corresponding to the feature (value).

[0113] Input the obtained key-value pairs and the real-time feature asset synchronization table (i.e., asset dictionary) into the UDF. According to the UDF, determine whether the key in the key-value pair is in the asset dictionary. And if the key is in the asset dictionary, obtain the key-value pairs of all real-time features (i.e., target features) of a log, and calculate the indicator value, such as whether it is called, etc., based on each real-time feature and its feature value under a log, and output the UDF.

[0114] For example, if the user searches on the homepage of the insurance sales interface, the corresponding feature value can be 1, and if not, the corresponding feature value can be 0. For the click sequence of each recommended insurance product on the homepage of the insurance sales interface, if the user clicks, there is a corresponding feature value, that is, the identifier of the insurance product clicked by the user.

[0115] Then, the indicator value of each key-value pair can be calculated: the proportion of null values / 0 values, and whether it is called. For example, for the sequence feature of the click sequence of each recommended insurance product on the homepage of the insurance sales interface, the proportion of null values / 0 values ​​can be calculated, and it can be determined whether the model has called the feature for each user, that is, whether the user has clicked on the insurance product.

[0116] Then, through the Structured Query Language (SQL) time window aggregation statistics, the call statistics within the preset time are obtained, and a summary table of the call statistics of the feature calls is obtained.

[0117] Next, we will explain how to calculate the above-mentioned various call statistics.

[0118] As for timeliness, for each user, the generation time of the corresponding content in the user behavior log can be obtained, and then the timeliness can be determined based on the difference between the generation time of the corresponding content in the user behavior log and the call time in the call log. Specifically, the average value of the above difference corresponding to each user can be used as the timeliness. For the convenience of calculation, when performing feature engineering, the timestamp of the user behavior corresponding to the feature can be generated when the feature engineering generates the corresponding feature; and when the model is called, the timestamp of the user behavior can be recorded in the call log, so as to facilitate the timeliness calculation.

[0119] As for the call volume, the index value of each key-value pair is calculated above, and in the case of a call, whether it is called is taken as 1. Then, for the feature key-value pairs in a specific time window, the values ​​of whether it is called can be summed to obtain the call volume. Correspondingly, the daily / real-time proportion can be determined based on the above call volume and the sum of the number of key-value pairs corresponding to all the target features.

[0120] For sequence features, the proportion of empty values / 0 values ​​can be determined based on the average value of the proportion of empty values / 0 values ​​in the indicator values ​​of each key-value pair to determine the proportion of empty values / 0 values ​​of the target feature.

[0121] For the calculation of the feature positive sample rate of the 0 / 1 type feature, we can first determine the number of users whose value is 1, and then determine the total number of users based on the call volume, and then calculate the feature positive sample rate based on the two.

[0122] The feature coverage is similar to the above. The feature coverage can be calculated by dividing the call volume by the number of key-value pairs corresponding to the target feature based on the call volume and the number of all key-value pairs corresponding to the target feature (which can be determined based on whether the indicator value is called). In the case of multiple sequence features, the ratio between the feature coverage of each sequence feature can also be calculated to obtain the uniformity of label distribution.

[0123] Feature relevance can be determined based on the relevant results recorded in the call log. For example, after the feature call is completed, the user's click rate for the recommended product is obtained, which can be obtained by: after the model calls two features, the predicted user click rate for the recommended product is obtained; and after the model calls any of the above two features, the predicted user click rate for the insurance product recommended to the user is obtained. The feature relevance is determined by the ratio of the two user click rates.

[0124] Finally, step 205 is executed to generate an alarm based on whether the indicator is abnormal.

[0125] Step 205: When the call statistic value exceeds or falls below the alarm threshold corresponding to the target feature, an alarm is issued.

[0126] Specifically, if the call statistics are too high or too low, it may indicate that there is a problem with the quality of the target feature and an alarm needs to be issued to repair the feature. Figure 1 Abnormal indicator alarm in .

[0127] Specifically, different call statistics have different alarm thresholds. Some call statistics exceed the alarm threshold, indicating a problem, while some call statistics are lower than the alarm threshold, indicating a problem. The specific problems are described in the previous section and will not be repeated here.

[0128] Next, the method for determining whether to issue an alarm will be described in detail.

[0129] Firstly, the method of this specification can use a combination of long and short cycles to issue an alarm.

[0130] Specifically, the alarm threshold includes a first threshold and a second threshold; the preset time period includes a first preset period and a second preset period. Step 205 specifically includes: taking the current time as the time end point of the first preset period and the second preset period, determining the call statistics within the first preset period and the second preset period; the second preset period is longer than the first preset period. When the difference between the call statistics within the first preset period and the call statistics within any first preset period exceeds the preset first threshold, it is determined to be an alarm; or when the difference between the call statistics at the start time and the end time of the second preset period exceeds the second threshold, it is determined to be an alarm.

[0131] That is, when the difference between the call statistics value in a short period (the first preset period) and the call statistics value in any previous first period is greater than the first threshold, it is determined that the characteristic fluctuation is too large, and there may be problems with the validity or statistical interval, which needs to be repaired.

[0132] Specifically, when making a judgment, the difference between the current short-cycle call statistics and each other short-cycle call statistics can be calculated. Alternatively, multiple short-cycle call statistics from the past can be randomly selected for comparison. In addition, in some cases, the call statistics are calculated based on multiple instantaneous call statistics. In this case, the average of two instantaneous call statistics can be compared with the past call statistics.

[0133] In addition, it is also possible to determine whether there is a problem of continuous decline or continuous rise based on the long cycle.

[0134] Specifically, Figure 4 The horizontal axis represents time, and the vertical axis represents the size of the call statistics value. Figure 4 The long period in is the period from the first point to the last point, and the short period is the period from the last two points. The average value in the short period is the position corresponding to the dark blue dotted line. The average value position in the long period is shown by the green dotted line. In an optional embodiment, Figure 4 Each point in the graph can represent the call statistics value within one day. The short period can be 2 days. Therefore, the call statistics value within 2 days can be obtained based on the average of the call statistics values ​​within two days. The long period can include 5 days.

[0135] Figure 4 There is a problem of continuous decline in call statistics in a long period. In this case, it may be judged only based on the short period, and the problem of large fluctuations may not be discovered. Therefore, this specification also makes a judgment based on the long period, and judges whether the difference between the first and last instantaneous call statistics of the long period (i.e., the second preset period) is greater than the second threshold value to judge whether there is a problem of continuous decline or continuous rise. The first threshold value and the second threshold value can be the same.

[0136] The short cycle limits the average value within the short cycle to not be significantly different from the average call statistics of other short cycles, ensuring that the overall call statistics fluctuate within a certain range. The long cycle limits the call statistics at the beginning and end of a cycle to not be significantly different, preventing the call statistics from continuously decreasing or continuously being generated. In this manual, discrimination thresholds can be set for both long and short cycles, and an alarm can be triggered if any indicator fluctuates beyond the threshold. The threshold setting method combining long and short cycles can avoid conditions such as gradient descent / ascent in a single direction, ensuring that the characteristic fluctuation amplitude does not change continuously and stably.

[0137] In addition, the magnitudes of the first threshold and the second threshold can be determined according to the application scenario.

[0138] In an optional implementation, the determination of the call statistics can be performed separately, such as a separate determination for the call volume and a separate determination for the feature coverage. In addition, different call statistics can be weighted summed, and the determination of whether to issue an alarm can be made based on the weighted summed call statistics.

[0139] In other words, the call statistics include at least two statistics; the method for determining whether the call statistics exceed or are lower than the alarm threshold includes: performing weighted summation on the results of at least two statistics to obtain a weighted statistics; and determining whether the weighted statistics exceed or are lower than the alarm threshold corresponding to the target feature.

[0140] This method considers the importance differences of different call statistics and performs weighted summation, which can better ensure the accuracy of discrimination.

[0141] In an optional implementation, the weighted sum can be directly calculated based on a fixed weight value. The weight value can be determined based on the application scenario and the importance of the call statistic itself.

[0142] In another optional implementation, the weighted summation is performed by an ordered weighted averaging (OWA)-operator optimized Analytical Hierarchy Process (AHP).

[0143] Specifically, OWA operator theory is a discrete data weighting method that can effectively weaken the influence of subjective extreme value deviation on weight accuracy; AHP aims to obtain the relative weights of different indicator factors (i.e., calling statistical values) by comparing them, and then determine the evaluation results. Generally speaking, there are subjective extreme value deviations in the artificial weighting of AHP. Combining OWA operator theory with AHP method can effectively correct the weights obtained by AHP method, comprehensively consider the influence of the relative importance of different indicators and the order of importance, and perform weighting and evaluation more comprehensively. Figure 5 As shown in the figure, the specific steps of OWA-AHP method are as follows:

[0144] 1) Construct a discriminant matrix. Compare each evaluation index (i.e., call statistical values, the same below, no further description) in pairs to determine their relative importance, and construct a discriminant matrix A of relative importance weights = (a ij ) n×n ,(i=1,2,…,n;j=1,2,…,n),where a ij It is the ratio of the importance of the i-th indicator to the j-th indicator among the n indicators.

[0145] 2) Calculate the initial weight vector. The weight vector refers to the relative weight of each factor in each discriminant matrix. The weight can be calculated by the accumulation method and normalized. The initial weight vector q is obtained by formula (1): i as follows:

[0146]

[0147] where b ij is the normalized value of the ratio of the importance of each indicator; u i It is the normalized sum of the importance ratios of indicators in each row.

[0148] 3) Calculate the weighted vector. Arrange the weights of the first indicator obtained by the AHP method in descending order, and record the arranged data as (p0, p1, p2, ..., p n-1 ), and calculate p according to formula (2) j The weight vector v j :

[0149]

[0150] in is the number of combinations of selecting j data from n-1 data.

[0151] 4) Calculate the absolute weight vector. Based on the above weight vector and rearranged data, the absolute weight vector of each indicator is obtained by formula (3):

[0152]

[0153] 5) Calculate the corrected weight vector. The corrected weight vector w of each indicator is finally determined by formula (4): ai :

[0154]

[0155] 6) Repeat steps (2)-(5) to finally obtain the corrected weight vector of each indicator, which is the output result of the OWA-AHP method.

[0156] like Figure 1 As shown in the figure, after the alarm is issued, feature repair and governance steps can also be performed. Specifically, feature adjustments can be made according to the abnormal conditions of indicators, such as modifying the time window, re-evaluating ROI, etc. For example, if there are large fluctuations in statistical values ​​of different periods, the time window can be changed to balance the statistical values ​​of different periods. In addition, for low ROI, if the feature benefits are significantly lower than expected, it can be taken offline or a new batch of features can be put online.

[0157] In addition, in the case where the system reports an error that causes a surge in the number of calls, for example, the system recognizes the tapping of multiple buttons as the tapping of a specific button, the system can be maintained to reduce the occurrence of the above problem.

[0158] Through the above method, a full life cycle feature quality assurance solution is proposed, including different stages corresponding to before the feature is launched (computing task development, feature configuration) and after the feature is launched (feature analysis, statistical indicator calculation, alarm threshold setting and feature repair), which can perform full-link quality assurance before and after feature development and launch.

[0159] Before feature development goes online, performance evaluation is performed to ensure benefits, online and offline consistency checks are performed to ensure data quality, and R&D specification constraints and feature asset synchronization are used to automatically manage real-time data assets and back up feature dictionaries to facilitate multi-party consumption and achieve more efficient, comprehensive, and standardized asset management.

[0160] After feature development is launched, different statistical indicator calculation methods are proposed for different feature types (sequences or 0 / 1 values). The statistical indicator values ​​in different time periods are calculated from multiple aspects such as feature timeliness, effectiveness, outliers, contribution, coverage, etc., making feature monitoring more comprehensive and more universal.

[0161] Considering the problem of setting the characteristic alarm threshold, it is proposed to set the alarm threshold by combining long and short cycles (single indicator abnormality judgment) and weighted accumulation of fluctuation values ​​(multiple indicator abnormality judgment). The long and short cycle thresholds are set based on the average value of the past period, and at the same time, they can effectively avoid abnormal situations such as the gradient decline of characteristic fluctuation values; the weighted accumulation of fluctuation values ​​uses the hierarchical analysis method to determine the weights of different statistical indicators, and calculates the weighted summation results based on their fluctuation values, which can effectively judge the abnormal changes of characteristics under a certain trend. Combining the two together, the resulting characteristic alarm threshold will be more reasonable and reliable.

[0162] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0163] like Figure 6 As shown, Figure 6 This is a block diagram of a feature quality alarm device according to an exemplary embodiment of the present specification, the device comprising:

[0164] A call log acquisition module 610 is used to acquire a call log, where the call log is a log of a feature behavior of a model calling feature engineering;

[0165] A call statistics acquisition module 620 is used to acquire, for a target feature, a call statistics value of the target feature within a preset time period from the call log, wherein the call statistics value is used to indicate a statistical value describing a behavior of the model calling the target feature within a preset time period, or a characteristic of the target feature called by the model within a preset time period;

[0166] The alarm module 630 is used to issue an alarm when the call statistic value exceeds or falls below the alarm threshold corresponding to the target feature.

[0167] In an optional embodiment, the call statistics include at least one of the following: the average delay time between the generation and calling of the feature value of the target feature, the call volume of the target feature within a preset time period, the proportion of the call volume of the target feature within the preset time period to the total call volume of all features, the proportion of the first feature value in the target feature within the preset time period to the total number of target features, and the degree of influence of the target feature on the result.

[0168] In an optional embodiment, when the target feature is a sequence feature, the first feature value is a null value or a 0 value; the call statistics also include: feature coverage and feature coverage uniformity, the feature coverage is used to indicate the proportion of the number of objects including the target feature in a preset time period to the total number of objects; the feature coverage uniformity is used to indicate the uniformity of feature coverage corresponding to different target features; when the target feature is a binary feature, the first feature value is any one of the feature values.

[0169] In an optional implementation, the called statistical value includes at least two statistical values; the alarm module 630 is specifically used to perform weighted summation on the results of at least two statistical values ​​to obtain a weighted statistical value; and determine whether the weighted statistical value exceeds or is lower than the alarm threshold corresponding to the target feature.

[0170] In an optional embodiment, the weighted summation is performed by an ordered weighted average OWA-operator optimized hierarchical chromatography AHP.

[0171] In an optional embodiment, the alarm threshold includes a first threshold and a second threshold; the preset time period includes a first preset period and a second preset period; the alarm module 630 is specifically used to determine the call statistics within the first preset period and the second preset period with the current time as the time end point of the first preset period and the second preset period; the second preset period is longer than the first preset period; when the difference between the average value of the call statistics within the first preset period and the average value of the call statistics within any first preset period exceeds the preset first threshold, it is determined to be an alarm; or, when the difference between the call statistics at the start time and the end time of the second preset period exceeds the second threshold, it is determined to be an alarm.

[0172] In an optional embodiment, a feature dictionary is preset, and the target feature is a feature included in the feature dictionary. The locking device also includes a target feature screening module 640 (not shown in the figure), which is used to determine whether the called feature in the call log is a target feature according to the feature dictionary.

[0173] In an optional implementation, the target feature is a real-time feature.

[0174] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0175] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0176] like Figure 7 As shown, Figure 7A hardware structure diagram of a computer device is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0177] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0178] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0179] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0180] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0181] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0182] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0183] The embodiments of this specification also provide a computer program product, which implements the above-mentioned feature quality alarm method when executed by a processor.

[0184] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a feature quality alarm method is implemented.

[0185] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0186] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0187] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0188] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A feature quality alarm method, comprising: Obtaining a call log, where the call log is a log of a feature behavior of a model calling feature engineering; For the target feature, obtaining a call statistic value of the target feature within a preset time period from the call log, wherein the call statistic value is used to indicate the behavior of the model calling the target feature within the preset time period, or the characteristics of the target feature called by the model within the preset time period; When the call statistic value exceeds or falls below the alarm threshold corresponding to the target feature, an alarm is issued.

2. The method according to claim 1, wherein: The call statistics include at least one of the following: the average delay time between the generation and calling of the feature value of the target feature, the call volume of the target feature within a preset time period, the proportion of the call volume of the target feature within the preset time period to the total call volume of all features, the proportion of the first feature value in the target feature in the total number of target features within the preset time period, and the degree of influence of the target feature on the result.

3. The method according to claim 2, wherein: In the case where the target feature is a sequence feature, the first feature value is a null value or a 0 value; the call statistics also include: feature coverage and feature coverage uniformity, the feature coverage is used to indicate the proportion of the number of objects including the target feature in the preset time period to the total number of objects; the feature coverage uniformity is used to indicate the uniformity of feature coverage corresponding to different target features; When the target feature is a binary feature, the first feature value is any one of the feature values.

4. The method according to claim 2, wherein: The call statistics include at least two statistics; Methods for determining whether the call statistics value exceeds or falls below the alarm threshold include: Performing weighted summation on the results of at least two statistical values ​​to obtain a weighted statistical value; Determine whether the weighted statistical value exceeds or is lower than the alarm threshold corresponding to the target feature.

5. The method according to claim 4, wherein: The weighted summation is accomplished by ordered weighted average OWA-operator optimized hierarchical chromatography AHP.

6. The method according to claim 1, wherein: The alarm threshold includes a first threshold and a second threshold; the preset time period includes a first preset period and a second preset period; The step of issuing an alarm when the calling statistic value exceeds or falls below the alarm threshold corresponding to the target feature includes: Taking the current time as the end point of the first preset period and the second preset period, determining the call statistics within the first preset period and the second preset period; the second preset period is longer than the first preset period; When the difference between the call statistics value within the first preset period and the call statistics value within any first preset period exceeds a preset first threshold, determining to issue an alarm; Or, when the difference between the call statistics at the start time and the end time of the second preset period exceeds the second threshold, it is determined to issue an alarm.

7. The method according to claim 1, wherein: A feature dictionary is preset, and the target feature is a feature included in the feature dictionary; The method further comprises: For the feature called in the call log, it is determined whether the called feature is a target feature according to the feature dictionary.

8. The method according to claim 1, wherein: The target feature is a real-time feature.

9. A characteristic quality alarm device, comprising: A call log acquisition module is used to acquire a call log, where the call log is a log of a feature behavior of a model calling feature engineering; A call statistics acquisition module, used for acquiring, for a target feature, a call statistics value of the target feature within a preset time period from the call log, the call statistics value being used to indicate a statistical value describing a behavior of the model calling the target feature within the preset time period, or a characteristic of the target feature called by the model within the preset time period; The alarm module is used to issue an alarm when the call statistic value exceeds or falls below the alarm threshold corresponding to the target feature.

10. A computer program product, wherein when the computer program product is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 8 by running the executable instructions.

Images