A feature warehouse feature value monitoring method and system
By analyzing the multidimensional feature performance of the feature repository and constructing a baseline model through self-supervised learning, the stability problem caused by the susceptibility of feature values to interference is solved, and automated monitoring and efficient anomaly detection of feature data are realized, thereby improving the robustness and accuracy of the machine learning system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI SHUOEN NETWORK TECH CO LTD
- Filing Date
- 2025-11-24
- Publication Date
- 2026-06-09
AI Technical Summary
Feature values in the feature warehouse are susceptible to factors such as data drift, acquisition noise, external event interference, or upstream pipeline failure, which can lead to imbalance in distribution stability, abnormal timestamp sequences, and consequently, a decrease in model prediction accuracy and distortion of system decision-making.
By performing multidimensional feature performance analysis on historical feature value groups, a feature performance baseline model is constructed. Review tags are obtained using a large language model, and distribution stability and timestamp distribution analysis are performed. Combined with self-supervised learning, a baseline parameter set is generated to achieve real-time multidimensional analysis and deviation detection of real-time feature value groups.
It enables automated monitoring of feature data, improves the robustness and anomaly detection accuracy of machine learning pipelines, provides low-latency, high-precision real-time deviation quantification, and avoids the rigid limitations of traditional thresholding methods.
Smart Images
Figure CN121579904B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of feature warehouse monitoring technology, and in particular to a method and system for monitoring feature values in a feature warehouse. Background Technology
[0002] In the fields of machine learning and artificial intelligence applications, feature warehouses serve as core infrastructure for data engineering, used to centrally store, manage, and distribute feature sets extracted from massive amounts of raw data. These feature sets are widely used in model training, online inference, and real-time decision-making scenarios. However, with the diversification of data sources and the explosive growth of business scale, feature values in feature warehouses are susceptible to factors such as data drift, acquisition noise, external event interference, or upstream pipeline failures. This can lead to an imbalance in the distribution stability of feature sets, abnormal timestamp sequences, or deviations in overall patterns, resulting in decreased model prediction accuracy, distorted system decisions, and even security risks. Summary of the Invention
[0003] The purpose of this invention is to provide a method and system for effectively monitoring feature values in a feature repository.
[0004] This invention discloses a method for monitoring feature values in a feature repository, comprising:
[0005] Step S100: Analyze the historical feature warehouse record data and obtain the review label for each historical feature value group. The review label is a description of the occurrence of the historical feature value group.
[0006] Step S200: Perform multidimensional feature performance analysis on the historical feature value group to determine the feature performance in different dimensions, obtain the multidimensional feature performance group, and construct the feature performance baseline model based on the multidimensional feature performance group and its corresponding review label to characterize the normal performance pattern of historical feature values.
[0007] Step S300: Perform real-time multidimensional analysis on the real-time feature value group and compare it with the data output by the feature performance baseline model to obtain the feature deviation index. Issue a data anomaly warning for the real-time feature value group whose feature deviation index is greater than or equal to the preset value.
[0008] In some embodiments disclosed in this invention, the method for obtaining the review label includes:
[0009] Step S101: Construct the label field and use the large language model to retrieve and analyze the historical data corresponding to the historical feature value group, retrieve the label content field that is synonymous with the label field, and associate it with the context description to form a structured label dataset.
[0010] In some embodiments disclosed in this invention, the multidimensional feature performance analysis includes: distribution stability analysis and timestamp distribution analysis.
[0011] In some embodiments disclosed in this invention, the method for performing multidimensional feature performance analysis on historical feature value groups includes:
[0012] Step S201, the method for performing distribution stability analysis on the historical eigenvalue set includes:
[0013] Step S2011: Classify the historical feature values in the historical feature value group based on the feature value type to obtain several historical feature value subgroups. Sort each historical feature value subgroup in chronological order to obtain a historical feature value subsequence.
[0014] Step S2012: Randomly select several historical feature value groups and randomly combine the historical feature value groups several times. Align the corresponding historical feature value subsequences between the historical feature value groups and determine the subsequence matching parameters between them. Based on the matching parameters of all subsequences between the historical feature value groups, determine the sequence group matching parameters between the historical feature groups. If the sequence group matching parameters are greater than or equal to a preset value, then the historical feature value groups are considered to match.
[0015] Step S2013: Analyze the matching historical feature value groups. If the same historical feature value group appears in several comparison combinations of historical feature values and the number of occurrences is greater than or equal to a preset value, then the corresponding historical feature value group is identified as a historical feature value group.
[0016] Step S2014: Compare different historical feature value groups with the historical feature value groups of the markers respectively, determine the sequence group matching parameters, and determine the matching group ratio of each historical feature value group corresponding to the matching historical feature value group. The matching group ratio is the ratio of the matching historical feature value group to all historical feature value groups.
[0017] Step S2015: The historical feature value group, the proportion of matching groups and the proportion of non-matching groups corresponding to each historical feature value group are used as the criteria for judging the distribution stability of the historical feature value group. The proportion of non-matching groups is the ratio of historical feature value groups without matching groups to all historical feature value groups.
[0018] The methods for determining the matching parameters between historical feature groups include:
[0019] Step S20121: Calculate the matching percentage of historical feature values in the historical feature value subsequences to all historical feature value subsequences, and define the matching percentage as the subsequence matching parameter. Determine the subsequence matching percentage when the subsequence matching parameter is greater than or equal to a preset value, and determine the matching parameter between historical feature value groups based on the preset matching percentage interval to which the subsequence matching percentage belongs.
[0020] In some embodiments disclosed in this invention, the method for performing multidimensional feature performance analysis on historical feature value groups includes:
[0021] Step S202, the method for performing timestamp distribution analysis on historical feature value groups includes:
[0022] Step S2021: Extract the timestamp associated with each historical feature value within the historical feature value group, calculate the statistical distribution of the timestamps within the group, including time mean, time skewness and time kurtosis, and generate a timestamp distribution parameter set;
[0023] Step S2022: Analyze the time difference characteristics between the timestamp distribution parameter set and the average baseline of historical feature warehouse record data, including the time mean difference, time skewness difference, and time kurtosis difference, and identify the time difference characteristics as the timestamp distribution analysis results.
[0024] In some embodiments disclosed in this invention, the method for constructing a feature representation baseline model includes:
[0025] Step S203: Use the historical feature value group, the corresponding review label, and the multi-dimensional feature performance group in the historical feature warehouse record data as the initial feature vector;
[0026] Step S204: Preprocess the input data using the fusion training framework, including standardizing and normalizing the historical feature value group and embedding and encoding the reviewed labels to form a joint input matrix, supporting multimodal data alignment.
[0027] Step S205: A self-supervised learning algorithm is used to perform unlabeled training on the joint input matrix to generate a baseline parameter set, including distribution parameters and time series parameters, so as to achieve a joint representation of distribution stability and timestamp distribution dimension.
[0028] Step S206: The stability of the baseline parameter set is evaluated through cross-validation mechanism, and the parameters are dynamically adjusted according to the supervision signal of the review label to achieve a baseline model output that supports dynamic threshold adaptation.
[0029] In some embodiments disclosed in this invention, a method for performing real-time multidimensional analysis on a real-time feature value set and comparing it with data output from a feature performance baseline model includes:
[0030] Step S301: Analyze the real-time feature warehouse recorded data using the feature performance baseline model, generate several sets of feature values for verification in real time, and output corresponding review labels.
[0031] Step S302: Filter the verification feature value group, select several verification feature value groups that are similar to the real-time feature value group, and determine the degree of agreement between the selected verification feature value group and the real-time feature value group.
[0032] Step S303: Based on the preset matching degree range to which the matching degree belongs, and whether the review labels of the verification feature value group and the real-time feature value group match, find the corresponding feature deviation index in the feature deviation index correspondence table.
[0033] In some embodiments disclosed in this invention, the expression for determining the degree of matching is:
[0034] ;
[0035] in, To determine the degree of agreement, For the first The weight coefficients corresponding to each feature value To preset the maximum eigenvalue difference, To verify the eigenvalue set and the real-time eigenvalue set, the first... The height matches the parameters. To verify the eigenvalue set and the real-time eigenvalue set, the first... The difference in eigenvalues, For the first The influence of the sub-matching parameter on the translation constant is determined based on the maximum peak difference of the corresponding eigenvalue. The maximum peak difference of the eigenvalue is the difference between the maximum and minimum values of the corresponding eigenvalue in the real-time eigenvalue group.
[0036] In some embodiments disclosed in this invention, a feature warehouse feature value monitoring system is also disclosed, comprising:
[0037] The first module is used to analyze the historical feature warehouse record data and obtain the review label for each historical feature value group. The review label is a description of the occurrence of the historical feature value group.
[0038] The second module is used to perform multidimensional feature performance analysis on historical feature value groups, determine the feature performance in different dimensions, obtain multidimensional feature performance groups, and construct feature performance baseline models based on multidimensional feature performance groups and their corresponding review tags to characterize the normal performance pattern of historical feature values.
[0039] The third module is used to perform real-time multidimensional analysis on the real-time feature value group and compare it with the data output by the feature performance baseline model to obtain the feature deviation index. It provides data anomaly warning for real-time feature value groups whose feature deviation index is greater than or equal to the preset value.
[0040] This invention discloses a feature value monitoring method and system for a feature repository, relating to the field of feature repository monitoring technology. The method includes: analyzing historical feature repository recorded data to obtain review tags for each historical feature value group, with each tag recording a description of the feature value group's occurrence; conducting multi-dimensional feature performance analysis on historical feature value groups to determine feature performance in different dimensions, forming a multi-dimensional feature performance group, and constructing a feature performance baseline model based on this group and its corresponding review tags to characterize the normal performance pattern of historical feature values; performing real-time multi-dimensional analysis on real-time feature value groups, comparing the data with the baseline model output, calculating a feature deviation index, and issuing a data anomaly warning for real-time feature value groups with deviation indices greater than or equal to a preset value. This invention achieves automated monitoring of feature data, improving the robustness and anomaly detection accuracy of machine learning pipelines.
[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0042] Figure 1 This is a flowchart illustrating the method steps of a feature warehouse feature value monitoring method disclosed in an embodiment of the present invention. Detailed Implementation
[0043] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0044] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. It should be understood that the preferred embodiments described herein are only for illustration and explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments based on the following content of the present invention. In the present invention, unless otherwise expressly specified and limited, the technical terms used in the present invention should have the ordinary meaning understood by those skilled in the art.
[0045] Example:
[0046] This invention discloses a method for monitoring feature values in a feature repository, see reference. Figure 1 ,include:
[0047] Step S100: Analyze the historical feature warehouse record data and obtain the review label for each historical feature value group. The review label is a description of the occurrence of the historical feature value group.
[0048] Step S100 obtains review tags for each historical feature value group by systematically analyzing the historical feature warehouse recorded data. These review tags essentially record the specific context in which the historical feature value group appeared, such as abnormal events, normal patterns, or external influencing factors, thus providing a semantic annotation basis for subsequent modeling. Specifically, this step first constructs a tag field, uses a large language model to retrieve and analyze historical data, identifies content fields synonymous with the tag field, and associates them with contextual descriptions to form a structured tag dataset. This method ensures the accuracy and comprehensiveness of the tags, avoids the bias of manual annotation, and injects natural language interpretive knowledge into the abnormal patterns of feature values, supporting the input of supervisory signals for subsequent multidimensional analysis. Ultimately, it realizes the semantic representation of historical data, laying a reliable data foundation for the baseline establishment of the entire monitoring system.
[0049] Step S200: Perform multidimensional feature performance analysis on the historical feature value group to determine the feature performance in different dimensions, obtain a multidimensional feature performance group, and construct a feature performance baseline model based on the multidimensional feature performance group and its corresponding review label to characterize the normal performance pattern of historical feature values.
[0050] Step S200 conducts a multidimensional feature performance analysis on historical feature value groups, aiming to extract feature performance from two dimensions: distribution stability and timestamp distribution. This generates multidimensional feature performance groups and, combined with their corresponding review tags, constructs a feature performance baseline model to characterize the normal performance patterns of historical feature values. Specifically, the distribution stability analysis categorizes historical feature values based on feature value type to form subsequences, performs random combination and alignment comparisons, calculates subsequence and sequence group consistency parameters, identifies historical feature value groups, and evaluates the proportion of consistent and non-consistent groups, thereby quantifying the stability between sequences. The timestamp distribution analysis extracts the time associated with each historical feature value. The system calculates the statistical distribution within a group, such as time mean, skewness, and kurtosis, and compares the temporal differences with the overall average baseline to capture temporal anomaly patterns. Subsequently, using historical feature value groups, multidimensional performance groups, and review labels as initial vectors, the system preprocesses, standardizes, and embeds the vectors to form a joint input matrix. Self-supervised learning is then used to generate a baseline parameter set, including distribution and temporal parameters. This set is dynamically adjusted through cross-validation and review label supervision to achieve a dynamically threshold-adaptive baseline model. Essentially, this model is a multimodal adaptive representation learning framework that can capture the implicit distribution patterns of normal modes, providing an interpretable benchmark reference for real-time monitoring.
[0051] Step S200 focuses on the quantitative evaluation and model fusion construction of multidimensional feature performance. By combining distribution stability analysis and timestamp distribution analysis, it extracts the inherent patterns of historical feature value groups to form a generalizable baseline model. This step first categorizes and sorts the historical feature value groups by type, performing random combination comparisons to calculate sequence group matching parameters and the proportion of matching groups, identifying key stable patterns. Simultaneously, it extracts statistical distribution parameters of timestamps (such as mean, skewness, and kurtosis) and compares them with the benchmark to generate time difference features. Subsequently, using the feature performance groups and corresponding review labels as input, it utilizes a fusion training framework for standardized preprocessing, self-supervised learning to generate parameter sets, and dynamically adjusts thresholds through cross-validation to achieve multimodal data alignment and joint representation. This adaptive modeling principle ensures that the baseline model not only captures the normal performance patterns of distribution and time series but also robustly copes with noise interference, providing an accurate reference benchmark for real-time deviation detection.
[0052] Step S300: Perform real-time multidimensional analysis on the real-time feature value group and compare it with the data output by the feature performance baseline model to obtain the feature deviation index. Issue a data anomaly warning for the real-time feature value group whose feature deviation index is greater than or equal to the preset value.
[0053] Step S300 performs real-time multidimensional analysis on the real-time feature value group and compares it with the data output by the feature performance baseline model to calculate the feature deviation index. An anomaly warning is issued for groups with deviations greater than or equal to a preset value, thereby realizing dynamic monitoring of the feature warehouse. Specifically, the baseline model analyzes the real-time recorded data to generate a verification feature value group and its review label. Then, a verification group that is similar to the real-time group is selected, and the degree of similarity between the two is calculated. Based on the interval of similarity and the consistency of the review label, the corresponding index is retrieved in the deviation index correspondence table. If the deviation exceeds the threshold, an alarm is triggered. This mechanism is essentially an anomaly detection framework based on similarity retrieval. Combined with the predictive ability of the baseline model, it achieves low-latency and high-precision real-time deviation quantification, avoiding the rigid limitations of traditional threshold methods and supporting robust monitoring of the feature warehouse in complex environments.
[0054] In some embodiments disclosed in this invention, the method for obtaining the review label includes:
[0055] Step S101: Construct the label field and use the large language model to retrieve and analyze the historical data corresponding to the historical feature value group, retrieve the label content field that is synonymous with the label field, and associate it with the context description to form a structured label dataset.
[0056] In some embodiments disclosed in this invention, the multidimensional feature performance analysis includes: distribution stability analysis and timestamp distribution analysis.
[0057] In some embodiments disclosed in this invention, the method for performing multidimensional feature performance analysis on historical feature value groups includes:
[0058] Step S201, the method for performing distribution stability analysis on the historical eigenvalue set includes:
[0059] Step S2011: Classify the historical feature values in the historical feature value group based on the feature value type to obtain several historical feature value subgroups. Sort each historical feature value subgroup in chronological order to obtain a historical feature value subsequence.
[0060] Step S2012: Randomly select several historical feature value groups and randomly combine them several times. Align the corresponding historical feature value subsequences between the historical feature value groups and determine the subsequence matching parameters between them. Based on the matching parameters of all subsequences between the historical feature value groups, determine the sequence group matching parameters between the historical feature groups. If the sequence group matching parameters are greater than or equal to a preset value, then the historical feature value groups are considered to match.
[0061] Step S2013: Analyze the matching historical feature value groups. If the same historical feature value group appears in several comparison combinations of historical feature values and the number of occurrences is greater than or equal to a preset value, then the corresponding historical feature value group is identified as a historical feature value group.
[0062] Step S2014: Compare different historical feature value groups with the historical feature value groups of the markers respectively, determine the sequence group matching parameters, and determine the matching group ratio of each historical feature value group corresponding to the matching historical feature value group. The matching group ratio is the ratio of the matching historical feature value group to all historical feature value groups.
[0063] Step S2015: The historical feature value group, the proportion of matching groups and the proportion of non-matching groups corresponding to each historical feature value group are used as indicators to judge the distribution stability of the historical feature value group. The proportion of non-matching groups is the ratio of historical feature value groups without matching groups to all historical feature value groups.
[0064] The methods for determining the matching parameters between historical feature groups include:
[0065] Step S20121: Calculate the matching percentage of historical feature values in the historical feature value subsequences to all historical feature value subsequences, and define the matching percentage as the subsequence matching parameter. Determine the subsequence matching percentage when the subsequence matching parameter is greater than or equal to a preset value, and determine the matching parameter between historical feature value groups based on the preset matching percentage interval to which the subsequence matching percentage belongs.
[0066] In some embodiments disclosed in this invention, the method for performing multidimensional feature performance analysis on historical feature value groups includes:
[0067] Step S202, the method for performing timestamp distribution analysis on historical feature value groups includes:
[0068] Step S2021: Extract the timestamp associated with each historical feature value within the historical feature value group, calculate the statistical distribution of the timestamps within the group, including time mean, time skewness and time kurtosis, and generate a timestamp distribution parameter set.
[0069] Step S2022: Analyze the time difference characteristics between the timestamp distribution parameter set and the average baseline of historical feature warehouse record data, including the time mean difference, time skewness difference, and time kurtosis difference, and identify the time difference characteristics as the timestamp distribution analysis results.
[0070] In some embodiments disclosed in this invention, the method for constructing a feature representation baseline model includes:
[0071] Step S203: Use the historical feature value group, the corresponding review label, and the multi-dimensional feature performance group in the historical feature warehouse record data as the initial feature vector.
[0072] Step S204: The input data is preprocessed using the fusion training framework, including standardizing and normalizing the historical feature value group and embedding and encoding the reviewed labels to form a joint input matrix, which supports multimodal data alignment.
[0073] Step S205: A self-supervised learning algorithm is used to perform unlabeled training on the joint input matrix to generate a baseline parameter set, including distribution parameters and time series parameters, thereby achieving a joint representation of distribution stability and timestamp distribution dimension.
[0074] Step S206: The stability of the baseline parameter set is evaluated through cross-validation mechanism, and the parameters are dynamically adjusted according to the supervision signal of the review label to achieve a baseline model output that supports dynamic threshold adaptation.
[0075] In some embodiments disclosed in this invention, a method for performing real-time multidimensional analysis on a real-time feature value set and comparing it with data output from a feature performance baseline model includes:
[0076] Step S301: Analyze the real-time feature warehouse recorded data using the feature performance baseline model, generate several sets of feature values for verification in real time, and output corresponding review labels.
[0077] Step S301 utilizes the constructed feature performance baseline model to perform real-time analysis on the recorded data of the real-time feature warehouse, generating several sets of verification feature values and outputting corresponding review labels in real time. This process is essentially the inference application stage of the baseline model. The model performs multi-dimensional projection and pattern matching on the input real-time data based on the distribution parameters and temporal parameters trained in history, simulating historical or synthetic verification sets that are potentially similar to the current real-time feature value sets. At the same time, semantic interpretation, such as abnormal patterns or contextual descriptions, is injected through the embedded review label generation mechanism to ensure that the generated verification sets not only represent normal variations numerically but also carry traceable label information, thereby providing a multimodal comparison benchmark for subsequent screening. This real-time generation mechanism relies on the low-latency inference path of self-supervised learning, avoiding the overhead of computation from scratch, and supporting efficient anomaly detection of the feature warehouse under high-frequency data streams.
[0078] Step S302: Filter the verification feature value group, select several verification feature value groups that are similar to the real-time feature value group, and determine the degree of agreement between the selected verification feature value group and the real-time feature value group.
[0079] Step S302 filters the generated feature value group for verification, selects several verification groups that are similar to the real-time feature value group, and calculates the degree of similarity between the two. This filtering process adopts a similarity measurement algorithm. First, candidate groups are initially filtered out by distance calculation in the multidimensional feature space (such as Euclidean distance or cosine similarity), and then refined into approximate groups. This calculation framework is essentially a hybrid similarity index that integrates sequence matching and weighted deviation. It can efficiently quantify semantic and structural affinity in multidimensional space, provide a quantitative basis for deviation assessment, and avoid the coarseness of simple threshold filtering.
[0080] Step S303: Based on the preset matching degree range to which the matching degree belongs, and whether the review labels of the verification feature value group and the real-time feature value group match, find the corresponding feature deviation index in the feature deviation index correspondence table.
[0081] Step S303 retrieves the corresponding feature deviation index from the predefined feature deviation index correspondence table based on the preset similarity interval to which the similarity degree belongs, and the similarity between the verification feature value group and the real-time feature value group. This mechanism maps continuous similarity scores to discrete intervals and uses label consistency judgment (such as semantic overlap or category matching) as a joint key value to locate the deviation score in the table. For example, high similarity and consistent labels correspond to low deviation, while label mismatch amplifies the deviation to reflect potential abnormal semantics. This correspondence table design is based on empirical statistics and model validation, ensuring the interpretability and operability of the index. Finally, the deviation index is output for threshold comparison to trigger early warning. Essentially, it is a rule-based decision engine that bridges the continuous prediction and discrete alarm logic of the model output, supporting the automated response of the feature warehouse and the traceability of manual review.
[0082] In some embodiments disclosed in this invention, the expression for determining the degree of matching is:
[0083] .
[0084] in, To determine the degree of agreement, For the first The weight coefficients corresponding to each feature value To preset the maximum eigenvalue difference, To verify the eigenvalue set and the real-time eigenvalue set, the first... The height matches the parameters. To verify the eigenvalue set and the real-time eigenvalue set, the first... The difference in eigenvalues, For the first The influence of the sub-matching parameter on the translation constant is determined based on the maximum peak difference of the corresponding eigenvalues. The maximum peak difference of the eigenvalues is the difference between the maximum and minimum values of the corresponding eigenvalues in the real-time eigenvalue set.
[0085] In some embodiments disclosed in this invention, a feature warehouse feature value monitoring system is also disclosed, comprising:
[0086] The first module is used to analyze the historical feature warehouse record data and obtain the review label for each historical feature value group. The review label is a description of the occurrence of the historical feature value group.
[0087] The second module is used to perform multidimensional feature performance analysis on historical feature value groups, determine the feature performance in different dimensions, obtain multidimensional feature performance groups, and construct feature performance baseline models based on multidimensional feature performance groups and their corresponding review tags to characterize the normal performance pattern of historical feature values.
[0088] The third module is used to perform real-time multidimensional analysis on the real-time feature value group and compare it with the data output by the feature performance baseline model to obtain the feature deviation index. It provides data anomaly warning for real-time feature value groups whose feature deviation index is greater than or equal to the preset value.
[0089] This invention discloses a feature value monitoring method and system for a feature repository, relating to the field of feature repository monitoring technology. The method includes: analyzing historical feature repository recorded data to obtain review tags for each historical feature value group, with each tag recording a description of the feature value group's occurrence; conducting multi-dimensional feature performance analysis on historical feature value groups to determine feature performance in different dimensions, forming a multi-dimensional feature performance group, and constructing a feature performance baseline model based on this group and its corresponding review tags to characterize the normal performance pattern of historical feature values; performing real-time multi-dimensional analysis on real-time feature value groups, comparing the data with the baseline model output, calculating a feature deviation index, and issuing a data anomaly warning for real-time feature value groups with deviation indices greater than or equal to a preset value. This invention achieves automated monitoring of feature data, improving the robustness and anomaly detection accuracy of machine learning pipelines.
[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for monitoring feature values in a feature repository, characterized in that, include: Step S100: Analyze the historical feature warehouse record data and obtain the review label for each historical feature value group. The review label is a description of the occurrence of the historical feature value group. Step S200: Perform multidimensional feature performance analysis on the historical feature value group to determine the feature performance in different dimensions, obtain the multidimensional feature performance group, and construct the feature performance baseline model based on the multidimensional feature performance group and its corresponding review label to characterize the normal performance pattern of historical feature values. Step S300: Perform real-time multidimensional analysis on the real-time feature value group and compare it with the data output by the feature performance baseline model to obtain the feature deviation index. Issue a data anomaly warning for the real-time feature value group whose feature deviation index is greater than or equal to the preset value. Methods for constructing feature representation baseline models include: Step S203: Use the historical feature value groups, their corresponding review tags, and multi-dimensional feature performance groups from the historical feature warehouse data as the initial feature vector, where the review tags are used to annotate the abnormal patterns and context descriptions of the historical feature value groups. Step S204: Preprocess the input data using the fusion training framework, including standardizing and normalizing the historical feature value group and embedding and encoding the reviewed labels to form a joint input matrix, supporting multimodal data alignment. Step S205: A self-supervised learning algorithm is used to perform unlabeled training on the joint input matrix to generate a baseline parameter set, including distribution parameters and time series parameters, so as to achieve a joint representation of distribution stability and timestamp distribution dimension. Step S206: The stability of the baseline parameter set is evaluated through cross-validation, and the parameters are dynamically adjusted according to the supervision signal of the review label to achieve a baseline model output that supports dynamic threshold adaptation; Methods for performing real-time multidimensional analysis on real-time feature sets and comparing them with data output from a feature performance baseline model include: Step S301: Analyze the real-time feature warehouse recorded data using the feature performance baseline model, generate several sets of feature values for verification in real time, and output corresponding review labels. Step S302: Filter the verification feature value group, select several verification feature value groups that are similar to the real-time feature value group, and determine the degree of agreement between the selected verification feature value group and the real-time feature value group. Step S303: Based on the preset matching degree range to which the matching degree belongs, and whether the review labels of the verification feature value group and the real-time feature value group match, find the corresponding feature deviation index in the feature deviation index correspondence table; The expression for determining the degree of fit is: ； Where W represents the degree of agreement, The weight coefficient corresponding to the i-th feature value is... To preset the maximum eigenvalue difference, To verify the i-th sub-matching parameter in the eigenvalue set and the real-time eigenvalue set, To verify the difference between the i-th feature value in the feature value set and the real-time feature value set, Let be the influence translation constant of the i-th sub-matching parameter. The influence translation constant is determined based on the maximum peak difference of the corresponding eigenvalues. The maximum peak difference of the eigenvalues is the difference between the maximum and minimum values of the corresponding eigenvalues in the real-time eigenvalue group.
2. The feature warehouse feature value monitoring method according to claim 1, characterized in that, Methods for obtaining review tags include: Step S101: Construct the label field and use the large language model to retrieve and analyze the historical data corresponding to the historical feature value group, retrieve the label content field that is synonymous with the label field, and associate it with the context description to form a structured label dataset.
3. The feature warehouse feature value monitoring method according to claim 1, characterized in that, Multidimensional feature performance analysis includes: distribution stability analysis and timestamp distribution analysis.
4. The feature warehouse feature value monitoring method according to claim 1, characterized in that, Methods for performing multidimensional feature performance analysis on historical feature sets include: Step S201, the method for performing distribution stability analysis on the historical eigenvalue set includes: Step S2011: Classify the historical feature values in the historical feature value group based on the feature value type to obtain several historical feature value subgroups. Sort each historical feature value subgroup in chronological order to obtain a historical feature value subsequence. Step S2012: Randomly select several historical feature value groups and randomly combine the historical feature value groups several times. Align the corresponding historical feature value subsequences between the historical feature value groups and determine the subsequence matching parameters between them. Based on the matching parameters of all subsequences between the historical feature value groups, determine the sequence group matching parameters between the historical feature groups. If the sequence group matching parameters are greater than or equal to a preset value, then the historical feature value groups are considered to match. Step S2013: Analyze the matching historical feature value groups. If the same historical feature value group appears in several comparison combinations of historical feature values and the number of occurrences is greater than or equal to a preset value, then the corresponding historical feature value group is identified as a historical feature value group. Step S2014: Compare different historical feature value groups with the historical feature value groups of the markers respectively, determine the sequence group matching parameters, and determine the matching group ratio of each historical feature value group corresponding to the matching historical feature value group. The matching group ratio is the ratio of the matching historical feature value group to all historical feature value groups. Step S2015: The historical feature value group, the proportion of matching groups and the proportion of non-matching groups corresponding to each historical feature value group are used as the criteria for judging the distribution stability of the historical feature value group. The proportion of non-matching groups is the ratio of historical feature value groups without matching groups to all historical feature value groups. The methods for determining the matching parameters between historical feature groups include: Step S20121: Calculate the matching percentage of historical feature values in the historical feature value subsequences to all historical feature value subsequences, and define the matching percentage as the subsequence matching parameter. Determine the subsequence matching percentage when the subsequence matching parameter is greater than or equal to a preset value, and determine the matching parameter between historical feature value groups based on the preset matching percentage interval to which the subsequence matching percentage belongs.
5. The feature warehouse feature value monitoring method according to claim 1, characterized in that, Methods for performing multidimensional feature performance analysis on historical feature sets include: Step S202, the method for performing timestamp distribution analysis on historical feature value groups includes: Step S2021: Extract the timestamp associated with each historical feature value within the historical feature value group, calculate the statistical distribution of the timestamps within the group, including time mean, time skewness and time kurtosis, and generate a timestamp distribution parameter set; Step S2022: Analyze the time difference characteristics between the timestamp distribution parameter set and the average baseline of historical feature warehouse record data, including the time mean difference, time skewness difference, and time kurtosis difference, and identify the time difference characteristics as the timestamp distribution analysis results.
6. A feature warehouse feature value monitoring system, characterized in that, A method for monitoring feature values in a feature repository as described in any one of claims 1-5, comprising: The first module is used to analyze the historical feature warehouse record data and obtain the review label for each historical feature value group. The review label is a description of the occurrence of the historical feature value group. The second module is used to perform multidimensional feature performance analysis on historical feature value groups, determine the feature performance in different dimensions, obtain multidimensional feature performance groups, and construct feature performance baseline models based on multidimensional feature performance groups and their corresponding review tags to characterize the normal performance pattern of historical feature values. The third module is used to perform real-time multidimensional analysis on the real-time feature value group and compare it with the data output by the feature performance baseline model to obtain the feature deviation index. It provides data anomaly warning for real-time feature value groups whose feature deviation index is greater than or equal to the preset value.