Information data-oriented data capitalization governance and intelligent quality monitoring method

Abstract

The invention discloses an intelligence data-oriented data capitalization management and quality intelligent monitoring method and system, and the method comprises the steps: taking a statistical window as a unit, and forming a continuously traceable feature vector; in a plurality of continuous statistical windows, the operation state characteristics of the same type of data objects are evaluated, through an automatically formed data asset model, the system can instantiate and adaptively adjust data quality rules, continuous data quality monitoring is realized based on historical operation characteristics, and abnormal states are recorded; and feeding back an abnormal state result to the data asset model, and driving the data asset model to perform hierarchical evolution. According to the method, the data asset model is automatically formed and updated along with the actual operation state, the data quality judgment threshold value is automatically calculated according to the historical statistical result of the data asset model, and the situation that a unified static rule is insufficient in applicability in different intelligence data scenes is avoided. The system forms a quality state index based on the anomaly intensity and the time dimension, improves the anomaly recognition accuracy and stability, and reduces the risk of false alarm and missing report.

Description

Technical Field

[0001] This invention relates to the field of intelligence data governance and data asset management technology, and in particular to a data assetization governance and quality intelligent monitoring method for intelligence data. Background Technology

[0002] Intelligence data, as a core resource for modern intelligence analysis, assessment, and services, typically originates from multiple heterogeneous information systems. Its data format is complex, encompassing structured, semi-structured, and a large amount of unstructured data. This type of data is characterized by its dispersed sources, complex structure, frequent updates, and high timeliness requirements. As the importance of data-driven decision-making becomes increasingly prominent, data is no longer merely a passive storage object but has evolved into a critical data asset with clearly defined value attributes and usage boundaries. Therefore, systematic asset modeling, governance, and quality assurance of intelligence data have become an indispensable technical foundation for supporting the stable operation and intelligent analysis of intelligence systems.

[0003] In existing technologies, data asset management and data quality control generally employ techniques centered on metadata management, manual rule configuration, or post-event quality sampling. One common approach involves manually defining a data asset catalog to register and manage data tables, fields, and their sources; another approach uses pre-defined, fixed data quality rules to verify indicators such as data integrity and consistency. These technologies are applicable to scenarios with stable structures and limited data scale. However, in intelligence data scenarios, due to the continuous expansion of data sources, frequent changes in data structure, and the evolution of data content over time, traditional methods relying on manual configuration and static rules are difficult to effectively adapt to complex operating environments.

[0004] In existing technologies, data asset management and data quality monitoring are often independent. Data assets exist only as static registration objects, and data quality monitoring results cannot be used to update data asset models, making it difficult to form a continuously evolving data governance mechanism. Furthermore, existing solutions generally suffer from delayed perception of data quality status and fragmented governance processes. Data quality rules are typically set manually at the initial stage of system construction, lacking a dynamic adjustment mechanism based on actual operational data, leading to rules gradually becoming ineffective or insufficiently covering the data. Simultaneously, in actual use, the quality problems of intelligence data often exhibit hidden and phased characteristics, making it difficult for traditional sampling inspections or periodic verification methods to detect potential risks in a timely manner, affecting the reliability of subsequent analysis results.

[0005] For example, invention application number 202411419369.6 discloses a data asset management and evaluation system and method based on data governance. This application can effectively analyze the real-time value status of each data in the database, and associate and combine them based on the value status of each data. It can autonomously and intelligently link and output related data, greatly improving the efficiency of data utilization, and enhancing the use value of linked data through the association data chain. However, this application also has the following problems: it is a static, post-evaluation analysis method, lacking dynamic modeling and feedback evolution mechanisms, and the automated formation of data assets.

[0006] For example, invention application number 202310704152.9 discloses a data governance quality monitoring method. This method can improve the analytical efficiency of analysts, save labor costs, increase work efficiency, and provide monitoring and early warning functions as well as analysis result push notifications, enabling real-time monitoring of quality issues in various enterprise data assets. However, this method also has limitations: it lacks dynamic modeling and feedback evolution mechanisms, and it doesn't automate the formation of data assets.

[0007] Therefore, in reality, there is a need for a data asset governance and quality intelligent monitoring method for intelligence data, to build an automatic data asset modeling mechanism based on data characteristics, and at the same time to establish a data quality rule generation and adjustment mechanism that matches the data operation status, so as to improve the accuracy and continuous effectiveness of data quality monitoring and ensure the quality controllability and value maximization of intelligence data throughout its entire life cycle. Summary of the Invention

[0008] To address the aforementioned problems, the present invention aims to provide a data asset governance and quality intelligent monitoring method for intelligence data. This method achieves automatic identification and dynamic modeling of intelligence data assets without significantly increasing the cost of manual intervention, thus forming a technical solution for continuous and intelligent monitoring of data quality.

[0009] This invention provides a data assetization governance and quality intelligent monitoring method for intelligence data.

[0010] The first aspect: a data asset governance and quality intelligent monitoring method for intelligence data, including: S1. Analyze the multi-source heterogeneous intelligence data to form a unified computable data object; using a preset statistical window as a unit, perform runtime feature quantification calculation on each type of data object to form a continuously traceable feature vector. S2. Within multiple consecutive statistical windows, evaluate whether the operational characteristics of the same type of data object are stable. When the operational characteristics meet the stability conditions within multiple consecutive windows, automatically trigger asset formation and generate a data asset model for that type of data. S3. Based on the data asset model, automatically instantiate the data quality rules that match it; S4. At the end of each statistical window, recalculate the running state characteristics of the data object in the current window and compare them with the instantiated data quality rules. Intelligently monitor the running quality of the data object and record any abnormal states in the comparison of the data object. S5. Feed back the operational quality and abnormal status results of data objects to the data asset model to drive the hierarchical evolution of the data asset model.

[0011] In one embodiment of the present invention, the running state characteristics include: Structural features: field set and data type.

[0012] Content statistical characteristics: mean, standard deviation, and missing rate of numerical fields; mean length and keyword frequency distribution of text fields; Behavioral characteristics: number of data updates and number of accesses; Among them, after the runtime feature quantization calculation, a continuously traceable feature vector is formed, which is written into the feature storage module at the end of the window, along with the window number, timestamp and sample size.

[0013] In one embodiment of the present invention, the stability of the runtime characteristics of the same type of data objects in step S2 is determined by constraining the cross-window changes of the feature vector, as follows: in, This represents the mean of a certain feature vector within window t. This represents the mean of a certain feature vector within window t-1. The standard deviation of the previous window. This is the adjustment coefficient.

[0014] In one embodiment of the present invention, the data asset model is configured with baseline parameters and stability level classification, wherein the baseline parameters are the weighted average of the statistical results of the most recent windows, and the baseline parameters include baseline parameters of structural features, content statistical features and behavioral features.

[0015] In one embodiment of the present invention, the data quality rules include: Integrity rules are expressed as: in, The baseline for the historical missing rate recorded in the data asset model, for Current window missing rate This is the parameter for tolerance to missing values; Content distribution consistency rules are expressed as follows: Among them, This serves as the baseline for the keyword frequency distribution recorded in the data asset model. The KL divergence is used to measure the difference in keyword frequency distribution within the current window. This refers to the tolerance parameter for keyword frequency distribution. Abnormal Behavior Rules: in, This serves as the baseline for access counts recorded in the data asset model. This represents the number of times the current window has been accessed. This is the tolerance parameter for the number of visits.

[0016] In one embodiment of the present invention, the Missing rate tolerance parameter The difference between the 95th percentile of the historical missing rate and the baseline mean is taken. Keyword frequency distribution tolerance parameter Take the maximum value or upper quantile value of the KL divergence in the historical window; Access frequency tolerance parameter The confidence interval is obtained by constructing a confidence interval based on the standard deviation of the number of visits.

[0017] In one embodiment of the present invention, when recording and comparing abnormal states in S4, the magnitude of the abnormal state and the number of windows for the duration of the abnormal state are calculated. A persistent window counter is maintained for each type of abnormal state to determine whether each type of abnormal state is an occasional event or a trend change, and to classify it as a short-term abnormality or a long-term abnormality.

[0018] In one embodiment of the present invention, the S5 process drives the hierarchical evolution of the data asset model. In the event of a short-term anomaly, the measure is to update the baseline parameters of the content; In the event of prolonged anomalies, measures include updating the content baseline parameters, adjusting the stability level, or triggering a recalculation of data quality rules.

[0019] In one embodiment of the present invention, after the stability level changes, different monitoring frequencies or statistical window lengths are switched.

[0020] The second aspect: a data asset governance and quality intelligent monitoring system for intelligence data, including: The data access and feature quantification module is used to analyze and perform runtime feature quantification calculations on the accessed multi-source heterogeneous intelligence data. The asset identification and modeling module is used to automatically identify the stability of the runtime characteristics of the same type of data objects and generate data asset models. The data quality rule instantiation module is used to automatically instantiate data quality rules based on the data asset model. The quality monitoring and anomaly calculation module is used to perform intelligent quality monitoring and anomaly state calculation for each type of data object. The data asset model feedback evolution module is used to drive the evolution of the data asset model based on monitoring results and abnormal states.

[0021] Third aspect: An electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method provided in the first aspect.

[0022] Fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0023] The beneficial effects of this invention are: 1. The method of this invention continuously quantifies and analyzes the structural characteristics, content statistical characteristics, and usage behavior characteristics of intelligence data during actual operation. Based on an operational stability determination mechanism, it automatically generates a data asset model, avoiding the problem of relying on manual configuration of data asset catalogs and asset boundaries in existing technologies. This achieves the automatic transformation of intelligence data from raw data sets into measurable data assets, significantly reducing the reliance on human experience and maintenance in the data governance process.

[0024] 2. The method of this invention uses the data asset model as the core object of system operation, and integrates data asset governance and data quality monitoring into the same technical framework for management. Through the structural information, statistical baseline parameters, and stability levels fixed in the data asset model, the system can uniformly drive the generation of data quality rules, quality status assessment, and data asset model evolution at the asset level. This fundamentally solves the problem of the separation between data asset management and data quality monitoring in existing technologies, and realizes the coordinated operation of data governance and quality monitoring.

[0025] 3. In the method of this invention, the data quality rules are no longer static rules pre-set manually, but are automatically instantiated based on historical statistical baseline parameters in the data asset model. The threshold of each data quality rule is calculated by the system according to the historical window statistical distribution and can be dynamically adjusted as the state of the data asset model changes. This ensures that the quality monitoring strategy matches the actual operating characteristics of specific data assets, thereby effectively avoiding the problem of unified rules failing in different data scenarios and enhancing the adaptability of quality monitoring.

[0026] 4. In the quality monitoring process, the method of this invention not only determines whether an anomaly has occurred, but also calculates and records the magnitude of the anomaly and the duration of the anomaly window, thereby distinguishing between short-term random fluctuations and long-term trend changes. Through a comprehensive evaluation of the magnitude and duration of the anomaly, the system can more accurately identify risk events that truly affect data quality, reduce false alarms and false negatives, improve the credibility and interpretability of quality monitoring results, and enhance the accuracy of anomaly identification.

[0027] 5. The method of this invention feeds back the quality monitoring results to the data asset model, enabling the dynamic evolution of the baseline parameters and stability level of the data asset model. The system can automatically adjust the parameters of the data asset model and data quality rules according to the anomaly type and persistence, so that the data governance strategy is updated with the actual changes in data, thereby avoiding the problems of data asset model rigidity and long-term invalidation of data quality rules, and enhancing the system's adaptability to the long-term evolution characteristics of intelligence data. Attached Figure Description

[0028] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the system of the present invention; Figure 3 This is a flowchart illustrating the principle of feedback evolution of the data asset model in this invention. Figure 4 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0029] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0030] Existing solutions for the governance and intelligent quality monitoring of intelligence data assets generally suffer from problems such as delayed perception of data quality status and fragmented governance processes.

[0031] To address the aforementioned problems, this invention discloses a data assetization governance and quality intelligent monitoring method for intelligence data. To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments.

[0032] Example 1:

[0033] This embodiment discloses a data assetization governance and quality intelligent monitoring system for intelligence data, such as... Figure 2As shown, it includes: a data access and feature quantification module, an asset identification and modeling module, a data quality rule instantiation module, a quality monitoring and anomaly calculation module, and a data asset model feedback evolution module, among which: The data access and feature quantification module is used to analyze and perform runtime feature quantification calculations on the accessed multi-source heterogeneous intelligence data.

[0034] The data access and feature quantification module parses the accessed intelligence data and uses the parsed structured intelligence entries or data tables as the smallest processing objects. Within a preset statistical window, it quantifies the operational status of the intelligence data to form a set of operational feature parameters.

[0035] Specifically, the data access and feature quantification module first determines the number of data entries participating in the calculation within the current statistical window, and then calculates structural feature parameters reflecting the stability of the data structure, content statistical feature parameters reflecting the distribution characteristics of the data content, and behavioral feature parameters reflecting the data usage.

[0036] Structural feature parameters include the field set and data type; content statistical feature parameters include the mean, standard deviation, and missing rate of numerical fields, as well as the mean length, keyword set, and frequency distribution of text fields; behavioral feature parameters include the number of data updates and accesses within the statistical window. When storing these feature parameters, a window number and timestamp are included for subsequent cross-window comparative analysis.

[0037] For example, in the context of open-source intelligence related to electronic warfare in the United States, the system continuously accesses electronic warfare equipment capability description entries parsed from public reports and press releases, and uses 24 hours as a statistical window to calculate the average text length, missing rate, and keyword frequency distribution of the capability description fields. At the same time, it records the number of times this type of intelligence is updated and accessed within the window, thereby forming an operational characteristic description within the window.

[0038] The asset identification and modeling module is used to automatically identify the stability of the runtime characteristics of the same type of data objects and generate data asset models.

[0039] The asset identification and modeling module compares and analyzes the operational characteristic parameters within multiple consecutive statistical windows. When the intelligence data meets the preset stability judgment conditions in terms of structural composition, content distribution, and behavioral characteristics, the system automatically identifies it as a type of data asset and generates the corresponding data asset model.

[0040] Specifically, the automatic stability identification is based on time series statistical features. The system determines whether the field structure remains consistent within multiple consecutive windows and constrains the cross-window changes of key operational feature vector indicators. When the change range of the feature vector indicators is always within the historical statistical fluctuation range and the fluctuation of behavioral features is within the interpretable range, the asset formation operation is triggered to form a data asset model.

[0041] After the data asset model is generated, the asset identification and modeling module solidifies the structural summary information, content statistical baseline parameters, and behavioral baseline parameters of this type of data object, and assigns corresponding stability levels for hierarchical control of subsequent quality monitoring strategies.

[0042] In the open-source electronic warfare intelligence scenario, the system found that the field structure of intelligence describing the capabilities of electronic jamming equipment remained consistent across multiple consecutive statistical windows, and the changes in text length and keyword frequency distribution were always within the historical fluctuation range, indicating stable access behavior. Based on this, the system automatically generated the corresponding data asset model and set its stability level to a high level.

[0043] The data quality rule instantiation module is used to automatically instantiate data quality rules based on the data asset model.

[0044] The data quality rule instantiation module takes the data asset model as the input data asset model and instantiates data quality rules that match the data asset based on the data structure information and statistical baseline parameters recorded in the data asset model.

[0045] The data quality rules are parameterized judgment conditions, and their thresholds are derived directly from historical statistical results in the data asset model, rather than being pre-configured manually.

[0046] Data quality rules include integrity rules, content distribution consistency rules, and behavior anomaly rules. Integrity rules are used to limit the missing rate from deviating from the historical level of the asset; content distribution consistency rules are used to constrain the degree of difference between the current window content distribution and the historical baseline distribution; behavior anomaly rules are used to determine whether there are abnormal fluctuations in access or update behavior. The threshold parameters in each rule are automatically calculated by the system based on the historical window statistical distribution and are dynamically adjusted as the data asset model status changes.

[0047] In the electronic warfare intelligence data asset model, the system instantiates missing rate thresholds, keyword frequency distribution difference thresholds, and normal access frequency ranges based on historical statistical results, so that the quality monitoring strategy is consistent with the historical operating characteristics of the asset itself.

[0048] The quality monitoring and anomaly calculation module is used for intelligent quality monitoring and anomaly calculation for each type of data object.

[0049] During the continuous operation of intelligence data, the quality monitoring and anomaly calculation module recalculates the operational characteristic parameters at the end of each statistical window and compares them with the baseline parameters in the data asset model.

[0050] When any type of data quality rule is triggered, the system records the anomaly type, anomaly magnitude, and anomaly duration window, and updates the data asset quality status indicators accordingly.

[0051] Anomaly amplitude is used to characterize the degree of deviation between the current operating status and the historical baseline, while the anomaly duration window number is used to distinguish between short-term fluctuations and trend changes. Through a comprehensive evaluation of the above information, the system forms a quantitative description of the current quality and health status of the data assets.

[0052] In the context of open-source electronic warfare intelligence, when the system observes a significant increase in the difference between the frequency distribution of keywords within a certain statistical window and the historical baseline distribution, exceeding the corresponding threshold, the system identifies the event as an abnormal content distribution and accordingly increases the quality risk level of the asset.

[0053] The data asset model feedback evolution module is used to drive the evolution of the data asset model based on monitoring results and abnormal states.

[0054] The data asset model feedback evolution module feeds back the data quality monitoring results to the corresponding data asset model. Based on the anomaly type, magnitude, and duration, it updates the statistical baseline parameters and stability level parameters in the data asset model and adjusts the judgment thresholds of subsequent data quality rules accordingly.

[0055] When anomalies are short-term disturbances, the system employs a smooth update strategy to correct baseline parameters. When anomalies persist or occur frequently, the system lowers the asset stability level and recalculates data quality rule parameters to ensure that the monitoring strategy aligns with the new operational state of the data. Through this mechanism, the continuous evolution of the data asset model and quality monitoring strategy is achieved.

[0056] Example 2

[0057] Based on the system of Embodiment 1, this embodiment discloses a data assetization governance and quality intelligent monitoring method for intelligence data, such as... Figure 1 As shown, the steps include: S1. The system analyzes the multi-source heterogeneous intelligence data to form a unified computable data object. Using a preset statistical window as a unit, the system performs runtime feature quantification calculation on each type of data object to form a continuously traceable feature vector. The system then accesses and analyzes the intelligence data source.

[0058] First, complete the unified processing of computable objects. For structured digital data, the table field is the smallest object; for unstructured text data, the parsed intelligence entry is the smallest object, and a field-based representation is established at the entry level (e.g., fields such as source time, entity capability description, etc.).

[0059] Then, the system uses a preset statistical window to quantify the runtime characteristics of each type of data object, forming a continuously traceable feature vector, which is used for subsequent asset stability judgment and quality judgment threshold generation.

[0060] Operational characteristics include structural features, content statistical features, and behavioral features. Structural features include the set of fields and data types. Content statistical features include the mean, standard deviation, and missing rate of numerical fields; the mean length and keyword frequency distribution of text fields; and behavioral features include the number of data updates and the number of visits.

[0061] Furthermore, the system not only collects the current value to form a continuously traceable feature vector and writes it to the feature storage module at the end of the window, but also includes the window number, timestamp, and sample size, thereby ensuring that subsequent calculations (such as mean, variance, and distribution difference) have statistical significance and can be compared across windows.

[0062] Furthermore, the system performs runtime feature quantification calculations. Within each statistical window, the system first determines the sample set, that is, determines the number of data entries participating in the statistics within the current window. .

[0063] Average length of text field , represented as: in, Let be the length of the i-th intelligence text in characters.

[0064] Standard deviation Used to characterize the dispersion of text length within the window, for subsequent stability determination; keyword frequency distribution. The missing rate was obtained by normalizing the frequency of occurrence of all text within the window after word segmentation or keyword extraction. Calculated as a percentage of entries with missing fields out of the total number of entries; behavioral parameters , The logs are automatically summarized by the log system when the window ends.

[0065] All feature vector parameters are written to the feature storage module at the end of the window, along with the window number and timestamp, as input for subsequent cross-window analysis.

[0066] For example, taking open-source intelligence describing the capabilities of US electronic warfare equipment as an example, the system accesses publicly available reports and press releases, and parses them into a set of entries. Each entry includes: equipment / project name, capability description text, time, and source. The system calculates the average length of the capability description text fields within a 24-hour window. Missing rate And extract the keyword set K and keyword frequency distribution. (For example, normalized counts for jamming, electronic, and attack). Simultaneously, the system records the number of times this type of entry is updated within the window. Number of times the search results were accessed This is to reflect the activity and intensity of data usage.

[0067] S2. Within multiple consecutive statistical windows, assess the stability of the operational characteristics of the same type of data objects. Once the operational characteristics meet the stability conditions within these windows, asset formation is automatically triggered, generating a data asset model for that type of data object. The system observes the stability of the operational characteristics of the same type of data objects across multiple consecutive statistical windows. Stability does not simply mean that the structure remains unchanged, but rather that the structure, content, and behavior remain within a statistically interpretable range of fluctuation.

[0068] Specifically, the system introduces a stability determination mechanism based on historical window statistics: when the field structure remains consistent within M consecutive windows and the cross-window changes of key operational feature vector indicators do not exceed the allowable range, the system triggers the asset formation action, generates a data asset model, and solidifies the operational baseline parameters of this type of data object.

[0069] Preferably, the system maintains a stability observation buffer for each type of candidate data object to store the runtime characteristics of the most recent M windows. The system only starts the stability determination logic when the buffer is full, in order to avoid misjudgment due to insufficient data.

[0070] The cross-window variation of the feature vector index is constrained in the following way: here This represents the mean of a certain feature vector index within window t (such as text length or a numerical field). The standard deviation of the previous window. This is the adjustment coefficient (used to control the allowable fluctuation scale).

[0071] The system limits normal fluctuations to within the range of historical statistical fluctuations, thus avoiding misjudging natural noise as structural changes. This process is not a single-point judgment, but requires that the inequality be satisfied across multiple consecutive windows before the system considers the content characteristics to be within an acceptable fluctuation range. The significance of this mechanism lies in transforming asset formation from an instantaneous event into a time-series-based state transition, thereby avoiding frequent generation and revocation of data asset models.

[0072] After the data asset model is generated, its baseline parameters are taken as a weighted average of the statistical results of the most recent several windows, rather than simply taking the value of the last window, in order to improve the robustness of the data asset model. For example... Figure 3 As shown, the data asset model should at least include: structural summary information (field sets and types), and content statistical baselines ( , , ) and behavioral baseline ( , The system provides a stability level for subsequent strategy stratification (e.g., assets with high stability can be subject to more sensitive anomaly detection strategies).

[0073] For example, in an electronic warfare open-source intelligence scenario, the system uses the length of the capability description text as one of the feature vector indicators for stability evaluation. If the field set remains unchanged for seven consecutive windows, and Always less than Meanwhile, if the keyword frequency distribution overlap remains high (e.g., the main keyword set is stable), the system will identify this type of data as an assetizable object, generate a data asset model A_EW_Asset_01, and set the stability level to High, indicating that the data asset has entered a stage where a strict quality baseline can be established.

[0074] S3. Based on the data asset model, automatically instantiate the data quality rules that match it.

[0075] Once the data asset model is formed, the system begins to shift the quality assessment from manual configuration to data asset model-driven assessment.

[0076] Specifically, the system does not maintain a fixed set of data quality rule templates. Instead, it derives rule thresholds from the baseline parameters of the data asset model, giving each data asset a set of parameterized judgment conditions consistent with its historical operational characteristics. The purpose of this is to make quality monitoring adaptive. Different data assets have different levels of missing data, distribution fluctuations, and access activity. Using a uniform threshold would lead to long-term false alarms or missed alarms for some assets.

[0077] In the present invention, the data quality rules include integrity rules, content distribution consistency rules, and behavior anomaly rules.

[0078] Among them, the integrity rule is used to restrict the missing rate from deviating from the historical level of the asset: in, It is the baseline of historical missing rate recorded in the data asset model (which can be the mean or quantile). The missing rate tolerance parameter is derived from the dispersion of the historical window missing rate (e.g., taking a multiple of the historical standard deviation or the upper quantile difference).

[0079] Content distribution consistency rules are used to limit the degree of drift in keyword frequency distribution: in This serves as the baseline for the keyword frequency distribution recorded in the data asset model. The KL divergence is used to measure the difference in keyword frequency distribution within the current window. This is the keyword frequency distribution tolerance parameter, automatically determined by the statistical range of historical window distribution differences. This rule is used to identify typical quality issues such as structural changes that cause the failure of content topic drift extraction strategies.

[0080] Behavioral anomaly rules are used to monitor for abnormal fluctuations in access or update behavior: in, This serves as the baseline for access counts recorded in the data asset model. This is the access tolerance parameter, which can be calculated from the standard deviation or confidence interval of historical access counts. The significance of this rule is that a sudden and abnormal increase or decrease in access count often indicates that the data is being referenced excessively or that data availability has decreased, leading to retrieval failures, thus requiring a corresponding quality check.

[0081] In the above rules, , , These constants are not manually specified, but are automatically calculated by the data asset model during its formation phase based on historical statistical distributions. For example: The difference between the 95th percentile of the historical missing rate and the baseline mean can be used; The maximum value or upper quantile value of the KL divergence can be taken from the historical window; Confidence intervals can be constructed based on the standard deviation of the number of visits.

[0082] The aforementioned thresholds tend to converge when the data asset stability level is high, and can be appropriately relaxed or tightened when the stability level decreases, in order to cooperate with subsequent governance strategies.

[0083] For example, for the data asset model A_EW_Asset_01, the system obtains from the historical window that the baseline missing rate is approximately 1.6%, and sets the missing rate based on historical fluctuations. Therefore, instantiate The system determines the integrity criteria; it also obtains an acceptable range from historical statistics on keyword frequency distribution differences and sets... ; behavioral side will Set to 65 times per window, and use historical fluctuation settings. Thus forming The behavior determination range is defined by the specific historical characteristics of the asset mentioned above, rather than by a general configuration.

[0084] S4. At the end of each statistical window, recalculate the runtime characteristics of the data object in the current window and compare them with the instantiated data quality rules. Intelligently monitor the runtime quality of the data object and record any abnormal states in the data object comparison.

[0085] The system performs a monitoring and evaluation at the end of each statistical window: recalculating the current window's... , and Parameters, and compare them one by one with the data quality rules that have been instantiated for this data asset.

[0086] When any data quality rule is triggered, the system not only records whether it is abnormal, but also calculates the magnitude of the abnormality (such as the proportion of missing data exceeding the limit, the difference between KL and the limit) and the number of abnormality duration windows, in order to distinguish between short-term jitter and structural degradation.

[0087] When a data quality rule is triggered, the system not only records whether a violation has occurred, but also calculates the magnitude of the violation. For example: Excessive missing rate: Abnormal content distribution magnitude: Meanwhile, the system maintains a continuous window counter for each type of anomaly to determine whether the anomaly is an occasional event or a trend change. This counter serves as an important basis for the subsequent evolution of the data asset model.

[0088] The system also maintains an asset quality status indicator, Q, which comprehensively expresses the current health of the assets. Q can be composed of weighted scores for multiple anomalies, such as assigning weights to anomalies in missing rates, keyword frequency distribution, and behavior, with the weights increasing as the duration of the anomaly increases. This indicator is used for subsequent feedback evolution decisions in S5.

[0089] For example, within a certain window, the system observed a significant shift in the frequency distribution of electronic warfare intelligence keywords: compared to the baseline, the frequency of "jamming" decreased significantly, and a batch of new unexpected keywords appeared, leading to... Exceeding the threshold The system records this event as an anomaly in content distribution, writes the excess range of 0.09 and its occurrence window number into the monitoring log, and increases the asset's quality risk score Q. If this anomaly occurs in multiple consecutive windows, it will be considered a signal of asset quality degradation.

[0090] S5. Feed back the operational quality and abnormal status results of data objects to the data asset model to drive the hierarchical evolution of the data asset model.

[0091] The system feeds monitoring results back to the data asset model, enabling the model to evolve as the data changes. The key here is not simply updating the baseline, but rather adopting different evolution strategies based on the type and persistence of anomalies.

[0092] When an anomaly is short-term and sudden and subsides within a short window, the system can use a smooth update to correct the baseline; when an anomaly persists or occurs frequently, the system lowers the asset stability level S and adjusts the subsequent data quality rule parameters accordingly to ensure that the monitoring strategy is consistent with the new state of the data.

[0093] The evolution of data asset models adopts a hierarchical strategy: For short-term anomalies, only the content baseline parameters are updated, and a small update step size is used; For long-term anomalies, in addition to updating baseline parameters, the stability level should be adjusted, and / or the data quality rules should be recalculated.

[0094] When the stability level changes, the system can switch between different monitoring frequencies or statistical window lengths to adapt to the new operating state.

[0095] By employing the above strategies, the system avoids the problem of a single anomaly causing the data asset model to fail, while ensuring that the data asset model can evolve with long-term data changes.

[0096] For example, regarding content baselines Updates can employ a progressive fusion approach based on window weights (avoiding a single anomaly from dragging the baseline off-track); for decreased stability levels, more stringent structural consistency checks or higher-frequency monitoring with shorter windows can be triggered. Through these mechanisms, the system achieves parameter-level closed-loop linkage between monitoring results, data asset models, and rule thresholds.

[0097] If the KL out-of-limit anomaly occurs for only 1–2 windows, the system treats it as a short-term disturbance and updates the content baseline using a smaller learning rate. and maintain Unchanged; if KL exceeds the limit for 5 consecutive windows, the system determines that the asset has entered a structural change phase, downgrades the stability level to Medium, and... Adjust to a more sensitive or stricter interval (depending on the governance strategy: whether to detect anomalies earlier or avoid continuous false alarms), and conduct joint checks on source distribution and resolution links to promote the implementation of governance actions.

[0098] Application examples: In the context of open-source electronic warfare intelligence in the United States, the system continuously accesses descriptions of electronic warfare equipment capabilities from publicly available reports and press releases.

[0099] The system uses a 24-hour statistical window and first calculates the missing rate for each window. Average text length Standard deviation Keyword frequency distribution and number of visits Perform quantification and recording (corresponding to S1).

[0100] The field structure remained stable and the eigenvector indices were satisfied during multiple consecutive windows. After the stability constraints are met, the system triggers asset formation, generates a data asset model A_EW_Asset_01, and solidifies the content baseline. Behavioral baseline With stability level (Corresponding to S2).

[0101] Subsequently, the system instantiates data quality rules based on this data asset model: missing rate threshold. Keyword frequency distribution drift threshold Access behavior range (Corresponding to S3).

[0102] In subsequent running windows, the system compares and judges the parameters of each window: when a certain window is observed... When the limit is exceeded, the system records the abnormal content distribution, the cumulative number of abnormal duration windows, and updates the quality status indicator Q (corresponding to S4).

[0103] When an anomaly persists, the system reports it to the data asset model, lowers the stability level, adjusts the distribution threshold parameter, and performs a gradual update. To adapt to the long-term evolution of data, the data asset model and data quality rules will eventually evolve in tandem (corresponding to S5).

[0104] Through the aforementioned closed loop, the system achieves the automatic transformation of open-source electronic warfare intelligence data from raw datasets into measurable data assets, and continuously, parameterized, and interpretable monitoring and governance of its quality status.

[0105] The present invention also provides an electronic device, Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory, for example, to execute the following method: S1. Analyze the multi-source heterogeneous intelligence data to form a unified computable data object; using a preset statistical window as a unit, perform runtime feature quantification calculation on each type of data object to form a continuously traceable feature vector. S2. Within multiple consecutive statistical windows, evaluate whether the operational characteristics of the same type of data object are stable. When the operational characteristics meet the stability conditions within multiple consecutive windows, automatically trigger asset formation and generate a data asset model for that type of data. S3. Based on the data asset model, automatically instantiate the data quality rules that match it; S4. At the end of each statistical window, recalculate the running state characteristics of the data object in the current window and compare them with the instantiated data quality rules. Intelligently monitor the running quality of the data object and record any abnormal states in the comparison of the data object. S5. Feed back the operational quality and abnormal status results of data objects to the data asset model to drive the hierarchical evolution of the data asset model.

[0106] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments, including, for example: S1. Analyze the multi-source heterogeneous intelligence data to form a unified computable data object; using a preset statistical window as a unit, perform runtime feature quantification calculation on each type of data object to form a continuously traceable feature vector. S2. Within multiple consecutive statistical windows, evaluate whether the operational characteristics of the same type of data object are stable. When the operational characteristics meet the stability conditions within multiple consecutive windows, automatically trigger asset formation and generate a data asset model for that type of data. S3. Based on the data asset model, automatically instantiate the data quality rules that match it; S4. At the end of each statistical window, recalculate the running state characteristics of the data object in the current window and compare them with the instantiated data quality rules. Intelligently monitor the running quality of the data object and record any abnormal states in the comparison of the data object. S5. Feed back the operational quality and abnormal status results of data objects to the data asset model to drive the hierarchical evolution of the data asset model.

[0108] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data asset governance and quality intelligent monitoring method for intelligence data, characterized in that, include: S1. Analyze the multi-source heterogeneous intelligence data that has been accessed to form a unified computable data object; Using a preset statistical window as a unit, the runtime feature quantification calculation is performed on each type of data object to form a continuously traceable feature vector; S2. Within multiple consecutive statistical windows, evaluate whether the operational characteristics of the same type of data object are stable. When the operational characteristics meet the stability conditions within multiple consecutive windows, automatically trigger asset formation and generate a data asset model for that type of data. S3. Based on the data asset model, automatically instantiate the data quality rules that match it; S4. At the end of each statistical window, recalculate the running state characteristics of the data object in the current window and compare them with the instantiated data quality rules. Intelligently monitor the running quality of the data object and record any abnormal states in the comparison of the data object. S5. Feed back the operational quality and abnormal status results of data objects to the data asset model to drive the hierarchical evolution of the data asset model.

2. The method according to claim 1, characterized in that, The operational characteristics include: Structural features: field set and data type; Content statistical characteristics: mean, standard deviation, and missing rate of numerical fields; mean length and keyword frequency distribution of text fields; Behavioral characteristics: number of data updates and number of accesses; Among them, after the runtime feature quantization calculation, a continuously traceable feature vector is formed, which is written into the feature storage module at the end of the window, along with the window number, timestamp and sample size.

3. The method according to claim 2, characterized in that, In step S2, the stability of the runtime characteristics of the same type of data objects is evaluated by constraining the cross-window changes of the feature vector, as shown below: in, This represents the mean of a certain feature vector within window t. This represents the mean of a certain feature vector within window t-1. The standard deviation of the previous window. This is the adjustment coefficient.

4. The method according to claim 3, characterized in that, The data asset model is configured with baseline parameters and stability level classification. The baseline parameters are the weighted average of the statistical results of the most recent windows. The baseline parameters include baseline parameters of structural features, content statistical features and behavioral features.

5. The method according to claim 1, characterized in that, The data quality rules include: Integrity rules are expressed as: in, The baseline for the historical missing rate recorded in the data asset model, for Current window missing rate This is the parameter for tolerance to missing values; Content distribution consistency rules are expressed as follows: Among them, This serves as the baseline for the keyword frequency distribution recorded in the data asset model. The KL divergence is used to measure the difference in keyword frequency distribution within the current window. This refers to the tolerance parameter for keyword frequency distribution. Abnormal Behavior Rules: A c u r r e n t ∈ A 0 - δ 3 A 0 + δ 3 in, This serves as the baseline for access counts recorded in the data asset model. This represents the number of times the current window has been accessed. This is the tolerance parameter for the number of visits.

6. The method according to claim 5, characterized in that, The Missing rate tolerance parameter The difference between the 95th percentile of the historical missing rate and the baseline mean is taken. Keyword frequency distribution tolerance parameter Take the maximum value or upper quantile value of the KL divergence in the historical window; Access frequency tolerance parameter The confidence interval is obtained by constructing a confidence interval based on the standard deviation of the number of visits.

7. The method according to claim 1, characterized in that, In S4, when recording and comparing abnormal states, the magnitude of the abnormal state and the number of windows in which the abnormal state persists are calculated. A persistent window counter is maintained for each type of abnormal state to determine whether each type of abnormal state is an occasional event or a trend change, and to classify it as a short-term abnormality or a long-term abnormality.

8. The method according to claim 7, characterized in that, The S5-driven hierarchical evolution of the data asset model: In the event of a short-term anomaly, the measure is to update the baseline parameters of the content; In the event of prolonged anomalies, measures include updating the content baseline parameters, adjusting the stability level, or triggering a recalculation of data quality rules.

9. The method according to claim 8, characterized in that, After the stability level changes, switch to different monitoring frequencies or statistical window lengths.

10. The method according to any one of claims 1 to 9, characterized in that, A data asset governance and quality intelligent monitoring system for intelligence data includes: The data access and feature quantification module is used to analyze and perform runtime feature quantification calculations on the accessed multi-source heterogeneous intelligence data. The asset identification and modeling module is used to automatically identify the stability of the runtime characteristics of the same type of data objects and generate data asset models. The data quality rule instantiation module is used to automatically instantiate data quality rules based on the data asset model. The quality monitoring and anomaly calculation module is used to perform intelligent quality monitoring and anomaly state calculation for each type of data object. The data asset model feedback evolution module is used to drive the evolution of the data asset model based on monitoring results and abnormal states.

Images