A data asset-based inventory method, system and storage medium

Abstract

This invention relates to the field of data management technology, and particularly to a method, system, and storage medium for data asset inventory. The method involves randomly selecting a first subject from multiple subjects, collecting a first dataset from this first subject, and using it as a cumulative dataset. A second dataset from a second subject is then collected. Based on the cumulative dataset, a second set of data value factors is analyzed from the second dataset. Based on the proportion of data from each subject within the cumulative dataset, at least one categorical data extraction coefficient is configured, and at least one categorical dataset is extracted. An anomaly classifier is constructed to classify multiple second datasets as anomalies, obtaining a second set of data credibility factors. Data with values ​​greater than or equal to the data filtering factor threshold is selected and added to the cumulative dataset. This process is repeated to obtain the cumulative datasets from all subjects, which serve as the data asset inventory result. This method achieves the technical effect of improving the efficiency and quality of data asset inventory.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a method, system and storage medium for inventorying data assets. Background Technology

[0002] In today's data-driven era, data assets have become a crucial strategic resource for organizations, and effective data asset inventory is fundamental to maximizing data value. With the rapid development of information technology, the amount of data in organizations is exploding, originating from a wide range of sources, involving multiple entities, and exhibiting complex and diverse data types. Against this backdrop, accurately identifying valuable information from massive amounts of data from multiple entities, eliminating useless and abnormal data, and achieving a comprehensive and accurate inventory of data assets has become a critical challenge for organizations. Traditional data inventory methods often lack objectivity, are not scientifically sound in their assessment of data value, may be subject to human bias in initial data selection leading to distorted subsequent analysis results, and rely heavily on single dimensions or subjective judgments, making it difficult to accurately quantify the actual value of data. These methods suffer from technical problems related to the efficiency and quality of data asset inventory. Summary of the Invention

[0003] This invention addresses the technical problems of low efficiency and quality in data asset inventory in existing technologies by providing a data asset-based inventory method, system, and storage medium.

[0004] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0005] In a first aspect, the present invention provides a data asset inventory method, comprising: randomly selecting a first subject from multiple subjects, collecting a first dataset from the first subject as a cumulative dataset, wherein the cumulative dataset is updated as the data of the collected subjects is traversed; continuing to collect a second dataset from a second subject, analyzing a second data value factor set of the second dataset based on the cumulative dataset; configuring at least one classification data extraction coefficient based on the proportion of data of each subject in the cumulative dataset, extracting at least one classification dataset, constructing an anomaly classifier, classifying multiple pieces of second data as anomalies, and obtaining a second data credibility factor set; calculating a second data filtering factor set based on the second data value factor set and the second data credibility factor set, filtering out data that is greater than or equal to the data filtering factor threshold, adding it to the cumulative dataset, and continuing to filter to obtain the cumulative dataset of all subjects, as the data asset inventory result.

[0006] Optionally, a first subject is randomly selected from multiple subjects, and a first dataset of the first subject is collected as a cumulative dataset. This includes: acquiring multiple subjects to be inventoried of data assets; selecting a first subject among the multiple subjects, and collecting the first dataset of the first subject as a cumulative dataset, wherein the cumulative dataset is updated as the data of the collected subjects is traversed.

[0007] Optionally, the second dataset of the second subject is collected, and the second data value factor set of the second dataset is analyzed based on the cumulative dataset. This includes: collecting the second dataset of the second subject, wherein the second dataset includes multiple second data points; and analyzing the second data value factors of the multiple second data points in the second dataset based on the cumulative dataset to obtain the second data value factor set.

[0008] Specifically, based on the cumulative dataset, analyzing the second data value factors of multiple second data points within the second dataset to obtain a set of second data value factors includes: calculating the similarity between each second data point in the second dataset and all datasets in the cumulative dataset, and calculating the mean to obtain a set of second data similarity; and calculating the set of second data value factors based on the second data similarity set.

[0009] Optionally, based on the proportion of data for each subject within the cumulative dataset, at least one classification data extraction coefficient is configured, at least one classification dataset is extracted, an anomaly classifier is constructed, and multiple pieces of second data are classified as anomalies to obtain a second data credibility factor set. This includes: calculating the proportion of data for each subject within the cumulative dataset as at least one classification data extraction coefficient; obtaining a preset data extraction quantity, and calculating at least one classification data extraction quantity in combination with the at least one classification data extraction coefficient; randomly extracting data for at least one subject from the cumulative dataset according to the at least one classification data extraction quantity to obtain at least one classification dataset; using the at least one classification dataset, constructing an anomaly classifier, and classifying the second data as anomalies to obtain a second data credibility factor set.

[0010] The process involves using at least one classification dataset to construct an anomaly classifier, classifying multiple second data points as anomalies to obtain a second data credibility factor set. This includes: combining the at least one classification dataset to obtain an anomaly classification dataset; randomly selecting first anomaly classification data; constructing a single-layer classification node, where the single-layer classification node performs binary classification on the input data and labels data as anomaly when it is classified as a single data point; continuing to construct multiple layers of classification nodes to obtain the anomaly classifier; inputting multiple second data points into the anomaly classifier; obtaining the number of node layers at which each second data point is classified as anomaly; calculating the ratio to the maximum number of node layers to obtain multiple second data credibility factors; and integrating these to obtain a second data credibility factor set.

[0011] Optionally, based on the second data value factor set and the second data credibility factor set, a second data screening factor set is calculated. Data with values ​​greater than or equal to the data screening factor threshold is selected and added to the cumulative dataset. Further screening is then performed to obtain a cumulative dataset for all subjects, which serves as the data asset inventory result. This process includes: calculating the second data screening factor set based on the second data value factor set and the second data credibility factor set; selecting second data with values ​​greater than or equal to the data screening factor threshold and adding them to the cumulative dataset; and continuing to collect and screen data from other subjects to obtain a cumulative dataset for all subjects, which serves as the data asset inventory result.

[0012] Secondly, the present invention provides an inventory system based on data assets, comprising:

[0013] The cumulative data acquisition module is used to randomly select a first subject from multiple subjects, collect the first dataset of the first subject as the cumulative dataset, and update the cumulative dataset as the data of the subjects are collected.

[0014] The value factor analysis module is used to continue collecting the second dataset of the second subject, and to analyze the second data value factor set of the second dataset based on the accumulated dataset.

[0015] The data credibility analysis module is used to configure at least one classification data extraction coefficient based on the proportion of data of each subject in the cumulative dataset, extract at least one classification dataset, construct an anomaly classifier, classify multiple second data into anomalies, and obtain a second data credibility factor set.

[0016] The cumulative data filtering module is used to calculate the second data filtering factor set based on the second data value factor set and the second data credibility factor set, filter out data that is greater than or equal to the data filtering factor threshold, add it to the cumulative dataset, and continue filtering to obtain the cumulative dataset of all subjects as the data asset inventory result.

[0017] Thirdly, this application provides a storage medium storing a second computer software program, which, when executed by a processor, implements a data asset-based inventory method as described in the first aspect.

[0018] By implementing this invention, it is possible to randomly select a first subject from multiple subjects, collect a first dataset from the first subject, and use it as a cumulative dataset. The cumulative dataset is updated as the data of the collected subjects is traversed. Randomly selecting the first subject can avoid the bias caused by human selection, making the initial cumulative dataset more objective and laying a fair foundation for subsequent data processing. The cumulative dataset is dynamically updated as the collection process is carried out, and new data can be integrated in real time to ensure that the dataset can reflect the data status of the collected subjects in a timely manner, providing the latest reference for subsequent analysis.

[0019] By implementing this invention, it is possible to continue collecting a second dataset from a second subject, and to analyze a second data value factor set of the second dataset based on the cumulative dataset. Since the cumulative dataset contains valuable data information collected previously, using the cumulative dataset as a reference to analyze the value factors of the second dataset can make the analysis results more targeted and relevant. Furthermore, by obtaining the second data value factor set through analysis, the value of each piece of second data can be quantified, providing an important basis for value judgment for subsequent data screening.

[0020] By implementing this invention, it is possible to configure at least one classification data extraction coefficient based on the proportion of data of each subject in the cumulative dataset, extract at least one classification dataset, construct an anomaly classifier, classify multiple second data points as anomalies, and obtain a set of credibility factors for the second data points. Configuring the extraction coefficient based on the proportion of subject data makes the extracted classification dataset more representative, reflects the weight of different subject data in the cumulative dataset, and improves the construction quality of the anomaly classifier. Constructing an anomaly classifier to classify the second data points as anomalies can effectively identify abnormal data, and the obtained set of credibility factors can quantify the credibility of each second data point, providing a reliability judgment standard for data screening.

[0021] By implementing this invention, a second data screening factor set can be calculated based on the second data value factor set and the second data credibility factor set. Data with values ​​greater than or equal to the data screening factor thresholds can be selected and added to the cumulative dataset. This process continues to select and obtain the cumulative dataset of all subjects, which serves as the data asset inventory result. By selecting factors based on both value and credibility dimensions, the data can be evaluated more comprehensively, ensuring that the data added to the cumulative dataset has both high value and high credibility, thus improving the quality of the cumulative dataset. By selecting all subjects and forming the inventory result, a systematic inventory of data assets can be achieved, ensuring the completeness and accuracy of the inventory result and providing a reliable basis for the management and utilization of data assets.

[0022] In summary, by implementing this invention, the technical effect of improving the efficiency and quality of data asset inventory can be achieved. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a data asset-based inventory method provided by this invention;

[0024] Figure 2 A schematic diagram of the structure of an inventory system based on data assets provided by the present invention;

[0025] Figure 3 This is a schematic diagram of a storage medium provided by the present invention.

[0026] In the attached diagram, the components represented by each number are as follows:

[0027] The system includes a cumulative data acquisition module 11, a value factor analysis module 12, a data credibility analysis module 13, a cumulative data filtering module 14, a storage medium 400, and a first computer program 410. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0030] In the description of this invention, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0031] Example 1, as Figure 1As shown, this embodiment of the invention provides a data asset-based inventory method, including:

[0032] In step S100 of this application embodiment, a first subject is randomly selected from multiple subjects, and a first dataset of the first subject is collected as a cumulative dataset, including:

[0033] Acquire multiple entities whose data assets are to be inventoried;

[0034] Select a first subject among the plurality of subjects, collect a first dataset from the first subject, and use it as a cumulative dataset, wherein the cumulative dataset is updated as the subject data is collected is traversed.

[0035] In this embodiment, the purpose of step S100 is to establish an objective and dynamic initial baseline for the entire data asset inventory process. On the one hand, by randomly selecting the first subject, the bias in the initial data caused by human selection is avoided, ensuring that the starting point of the cumulative dataset is representative and reducing errors caused by initial data deviation in subsequent analysis. On the other hand, it is clear that the cumulative dataset is continuously updated as new subject data is collected, enabling the dataset to integrate new information in real time, providing a continuously improving reference for subsequent value assessment and anomaly detection of other subject data.

[0036] Suppose a factory has 100 production machines of the same model, i.e., the multiple entities mentioned above. The operating data of each machine, such as temperature, speed, energy consumption, fault codes, etc., are data assets. This data can be used to train equipment operation and maintenance models, or even sold as sample data to companies in the same industry, and has certain commercial value. Therefore, it is necessary to conduct a data asset inventory.

[0037] First, we need to obtain information on the multiple entities for which the data assets to be inventoried are to be acquired. Let's assume that these multiple entities are 100 production machines in a factory, numbered 1-100.

[0038] Then, it is necessary to select the first subject among the multiple subjects, and collect the first dataset of the first subject as the cumulative dataset. The cumulative dataset is updated as the subject data is collected. In the example above, the first subject is randomly selected from 100 devices using a random number generator or similar method. Let's assume that device 37 is randomly selected.

[0039] Then, the first dataset of device 37 is collected, which includes all its operating data over the past 3 months, such as 2,000 temperature records, 500 energy consumption data, and 20 fault codes. This data is packaged into "Cumulative Dataset V1.0", which is the initial cumulative dataset.

[0040] Then continue to collect datasets from other entities such as device 89 and device 5, repeating the above process. Each time the data from a new device is filtered, the parts that meet the criteria are added to the cumulative dataset to keep it updated, such as V3.0, V4.0, etc., eventually covering the effective operating data of all 100 devices.

[0041] In step S200 of this application embodiment, the second dataset of the second subject is collected, and the second data value factor set of the second dataset is analyzed based on the accumulated dataset, including:

[0042] Collect a second dataset for the second subject, wherein the second dataset includes multiple pieces of second data;

[0043] Based on the cumulative dataset, analyze the second data value factors of multiple second data points within the second dataset to obtain a set of second data value factors.

[0044] In this embodiment, the core purpose of collecting the second dataset from the second subject is to provide specific analytical objects for subsequent data value analysis. It is a crucial link connecting the cumulative dataset and the second data value factor set. Specifically, based on the data of the first subject, data from the second subject is collected, gradually covering more subjects to provide a foundation for a comprehensive inventory of data assets and avoid incomplete inventory due to missing subjects. The multiple second data points contained in the second dataset are the direct objects for subsequent calculation of the "second data value factor set." Only by clearly identifying these multiple data points can the value of each data point be analyzed based on the cumulative dataset, achieving a refined evaluation of the second subject's data.

[0045] First, it is necessary to determine the second subject. Among the multiple subjects that have been identified for the data asset inventory, the second subject following the first subject is selected. For example, in the case of 100 devices, after the randomly selected device 37, device 15 is selected as the second subject.

[0046] Then, all relevant data generated by the second entity are collected to form a second dataset. Taking a device as an example, the second dataset may include multiple operational data points of device 15, such as temperature data at different time points, such as "30℃" and "32℃"; energy consumption data, such as "100kWh" and "105kWh"; and fault codes, such as "E01" and "E02". Each data point is an independent second dataset.

[0047] During the collection process, it is necessary to ensure that the second dataset covers multiple categories and multiple data points of the subject, rather than a single data point or partial data, so as to ensure that the subsequent value analysis can cover the full picture of the second subject's data.

[0048] For example, in the above example, the first subject is device 37, and the cumulative dataset contains 2,000 temperature records and 500 energy consumption records over 3 months. The second subject selects device 15 and collects its 3 months of operating data, which is the second dataset, including 1,800 temperature records, 450 energy consumption records, and 15 fault codes. These individual temperature, energy consumption, and fault code records are all considered "second data".

[0049] These multiple sets of secondary data will serve as the basis for subsequent calculations and accumulation of similarity and data value factors of the dataset. For example, the similarity between a certain "E01" fault code of device 15 and a similar code in the data of device 37 will be analyzed to assess its data value.

[0050] In step S200 of this application embodiment, based on the accumulated dataset, the second data value factors of multiple second data points within the second dataset are analyzed to obtain a set of second data value factors, including:

[0051] Calculate the similarity between each second data point in the second dataset and all datasets in the cumulative dataset, and calculate the mean to obtain the second data similarity set;

[0052] Based on the second data similarity set, the second data value factor set is calculated.

[0053] The core purpose of calculating the second set of data value factors in this step is to quantitatively evaluate the relative value of each data point in the second dataset, providing a judgment standard based on data value dimensions for subsequent data screening. The logic is that the lower the similarity between the second data point and the cumulative dataset, the more new information or special features it contains, and the higher its potential value; conversely, high similarity may indicate duplicate or redundant data, with lower value. The mean similarity value objectively reflects the uniqueness of each second data point. Furthermore, transforming the similarity set into a set of value factors converts the abstract concept of data value into concrete numerical values, making the values ​​of different data points comparable and screenable, providing a clear basis for subsequent comprehensive credibility factor-based data evaluation.

[0054] First, it is necessary to calculate the similarity between each second data point in the second dataset and all data points in the cumulative dataset. That is, for each second data point in the second dataset, it is necessary to calculate its similarity with all data points in the cumulative dataset. For example, if the second data point is the device's "temperature 35℃ + speed 1500r / min", then it is necessary to calculate the similarity with all temperature and speed combinations of device 37 in the cumulative dataset.

[0055] For example, if a data point from device 15 showing "E01 fault + temperature 40℃" has a similarity of 30% with 100 related fault data points in the cumulative dataset of device 37, then that data point has a similarity of 30% and is included in the similarity set. The similarity between each data point in the second dataset and all data points in the cumulative dataset is calculated, and the average is also calculated to obtain the second dataset similarity set.

[0056] For example, the specific similarity calculation method can be to first decompose each data point into two key feature dimensions: the fault code as a categorical variable and the temperature as a numerical variable. The temperature needs to be standardized according to the normal operating temperature range of the equipment. Assuming the normal operating temperature range is 30℃~50℃, the temperature standardization formula is: Standardized Temperature = (Actual Temperature - 30) / (50 - 30). Therefore, the standardized temperature of equipment 15 is (40 - 30) / (50 - 30) = 0.5. Further, the temperature difference between the two data points needs to be calculated, i.e., the absolute difference in standardized temperatures. Assuming the cumulative data used for calculation here is fault E01 + temperature 38℃, its standardized temperature is 0.4, and the temperature difference between the two data points is |0.5 - 0.4| = 0.1.

[0057] For fault codes, the matching degree can be used as a measure. If the fault codes of the two sets of data are the same, that is, both are E01, then the matching degree is 1; otherwise, it is 0.

[0058] Furthermore, based on the degree of reliance on the fault code matching degree and the temperature difference during actual implementation, weights W1 and W2 are assigned to the fault code matching degree and the temperature difference, respectively. The similarity can be obtained by weighted summation. The sum of W1 and W2 is 1. That is, similarity = W1 × fault code matching degree + W2 × (1 - |temperature difference|). Assuming W1 is 0.7 and W2 is 0.3, then similarity = 0.7 × 1 + 0.3 × (1 - 0.1) = 0.7 + 0.27 = 0.97, or 97%.

[0059] Then, based on the second data similarity set, the average similarity is converted into a value factor using a preset formula. Generally, similarity and value factor are negatively correlated: the lower the similarity, the higher the value factor. For example, value factor = 1 - average similarity; if the average similarity is 30%, then the value factor is 70%. Then, the value factors of all second data are integrated into a second data value factor set, completing the value quantification of each piece of second data.

[0060] In step S300 of this application embodiment, based on the proportion of data for each subject in the accumulated dataset, at least one classification data extraction coefficient is configured, at least one classification dataset is extracted, an anomaly classifier is constructed, and multiple pieces of second data are classified as anomalies to obtain a second data credibility factor set, including:

[0061] Calculate the proportion of data for each subject within the cumulative dataset, and use it as at least one categorical data extraction coefficient;

[0062] Obtain the preset data extraction quantity, and calculate the data extraction quantity of at least one category by combining the at least one category data extraction coefficient;

[0063] According to the number of data extracted for at least one category, data of at least one subject is randomly extracted from the cumulative dataset to obtain at least one category dataset.

[0064] Using the at least one classification dataset, an anomaly classifier is constructed to classify the second data as anomalies, thereby obtaining a second data confidence factor set.

[0065] In this embodiment of the application, the purpose of this step is to construct representative training samples for the anomaly classifier, laying the foundation for subsequent identification of the anomalies in the second data and calculation of the confidence factor.

[0066] The logic is as follows: by configuring the extraction coefficient based on the proportion of subject data, the extracted classification dataset reflects the weights of different subjects in the cumulative dataset. For example, if a subject has a high proportion of data, its sample size in the classification dataset will also be larger, avoiding bias in the anomaly classifier's identification of specific subject data due to sample bias. Furthermore, by calculating the classification data extraction quantity in conjunction with the preset data extraction quantity, the size of the classification dataset can be controlled while ensuring sample representativeness. This avoids insufficient generalization ability of the classifier due to too few samples, and excessive computational cost due to too many samples, ensuring the efficiency and accuracy of the anomaly classifier construction.

[0067] First, it is necessary to calculate the proportion of data for each subject within the cumulative dataset, which serves as at least one classification data extraction coefficient. Specifically, it is necessary to calculate the proportion of data for each subject in the cumulative dataset relative to the total data volume, i.e., the subject data proportion, and directly use this proportion as the classification data extraction coefficient for the corresponding subject.

[0068] For example, if the cumulative dataset contains 1000 data points from device 37 and 500 data points from device A, with a total of 1500 data points, then the sampling coefficient for device 37 is approximately 66.7% (1000 / 1500) and the sampling coefficient for device A is approximately 33.3% (500 / 1500).

[0069] Next, a preset data extraction quantity needs to be obtained. Combined with the at least one classification data extraction coefficient, at least one classification data extraction quantity is calculated. First, the preset total data extraction quantity needs to be determined; for example, to build an anomaly classifier, it is planned to extract 300 data points as samples from the cumulative dataset. Then, the preset total quantity is multiplied by the extraction coefficient for each subject to obtain the classification data extraction quantity for that subject, i.e., the number of samples that should be extracted for that subject.

[0070] For example, if the preset data extraction quantity is 300 data points, then the data extraction quantity of device 37 = 300 × 66.7% ≈ 200 data points, and the data extraction quantity of device A = 300 × 33.3% ≈ 100 data points.

[0071] Finally, according to the number of data extracted for at least one category, data for at least one subject is randomly extracted from the cumulative dataset to obtain at least one category dataset.

[0072] For example, based on the calculated number of classified data extracted, a corresponding number of data are randomly extracted from each subject data in the cumulative dataset. For instance, 200 data are randomly extracted from 1000 data from device 37, and 100 data are randomly extracted from 500 data from device A.

[0073] All extracted data are integrated to form a classification dataset, i.e., a set of 300 data points in the example above, which will be used to build an anomaly classifier later.

[0074] In step S300 of this application embodiment, an anomaly classifier is constructed using the at least one classification dataset to classify multiple pieces of second data as anomalies, thereby obtaining a set of second data credibility factors, including:

[0075] Combine the at least one classification dataset to obtain an anomaly classification dataset, randomly select the first anomaly classification data, and construct a layer of classification nodes, wherein the layer of classification nodes performs binary classification on the input data, and labels the data as anomaly data when the data is classified as a single data;

[0076] Continue building to obtain multi-layer classification nodes and obtain an anomaly classifier;

[0077] Multiple pieces of second data are input into the anomaly classifier to obtain the number of node layers at which each piece of second data is classified as an anomaly. The ratio of this number to the maximum number of node layers is calculated to obtain multiple second data credibility factors. These factors are then integrated to obtain a set of second data credibility factors.

[0078] In this embodiment of the application, the purpose of classifying multiple pieces of second data as anomalies to obtain a set of second data credibility factors is to quantitatively evaluate the credibility of each piece of data in the second subject and provide a judgment standard based on the "reliability" dimension for subsequent data screening.

[0079] The logic is as follows: an anomaly classifier is constructed through multi-layered classification nodes, gradually refining the anomaly judgment of data, improving the accuracy of anomaly data identification, and avoiding misjudgments caused by the one-sidedness of single-layer classification. The credibility of each data point is quantified by the ratio of the node layer at which data is judged as anomaly to the maximum number of layers. For example, data judged as anomaly only at deeper nodes has a higher credibility factor, making the credibility index quantifiable and comparable, and providing a reliable basis for subsequent comprehensive value factor screening of data.

[0080] First, it is necessary to combine at least one classification dataset to obtain an anomaly classification dataset, randomly select the first anomaly classification data, and construct a layer of classification nodes.

[0081] That is, combine at least one previously extracted classification dataset to form an anomaly classification dataset, for example, integrate the classification data of device 37 and device 15 into a set containing 300 data points.

[0082] Randomly select one data point from the anomaly classification dataset as the first anomaly classification data point, and build a classification node based on this. This node performs binary classification on the input data. For example, if the first anomaly classification data point is temperature 35℃, the first-level node may classify data with "temperature ≥ 35℃" into one category and "temperature < 35℃" into another category. When a data point becomes a "single data point" after classification, that is, it cannot be classified into the same category as other data, it is directly labeled as an anomaly data point.

[0083] Based on the classification results of the first-level classification node, the process of randomly selecting a reference data and constructing the next-level classification node is repeated for each type of data. For example, for the classification dataset with temperature ≥35℃ in the first-level node, a data point with temperature 40℃ is randomly selected as the reference to construct a second-level classification node. This data is further divided into two categories: temperature ≥40℃ and temperature <40℃. Similarly, isolated individual data points after classification are labeled as anomalies.

[0084] Repeat the above process until the preset maximum number of layers is reached to form an anomaly classifier containing multiple layers of classification nodes. The more layers, the more refined the classification. For example, the preset maximum number of layers can be 5.

[0085] Then, each piece of data in the second dataset is input into the anomaly classifier to track its classification path at each layer node. If a piece of data is determined to be a single piece of data, i.e., an anomaly, at the 3rd layer node, then the number of node layers at which it is classified as an anomaly is 3.

[0086] The ratio of the number of node layers classified as anomalous data to the maximum number of node layers is used as the confidence factor for that data. For example, if the maximum number of node layers is 5, the confidence factor for the second data is 3 / 5 = 0.6. The higher this ratio, the more layers of classification the data undergoes before it is judged as anomalous, and the higher its confidence level, meaning it is more likely to be normal data; conversely, the lower the ratio, the lower the confidence level.

[0087] Finally, the confidence factors of all the second data are calculated to form a set of confidence factors for the second data.

[0088] The above classification nodes can be constructed using a multi-level binary tree. Each level of the multi-level binary tree performs binary classification on the data. The specific method for classifying data using a multi-level binary tree is existing technology and will not be elaborated here.

[0089] In step S400 of this application embodiment, a second data screening factor set is calculated based on the second data value factor set and the second data credibility factor set. Data with values ​​greater than or equal to the data screening factor threshold is selected and added to the cumulative dataset. Further screening is then performed to obtain the cumulative dataset of all subjects, which serves as the data asset inventory result. This includes:

[0090] The second data screening factor set is calculated based on the second data value factor set and the second data credibility factor set.

[0091] Filter the data to obtain second data whose second data filtering factor is greater than or equal to the data filtering factor threshold, and add them to the cumulative dataset;

[0092] Continue collecting and filtering data from other entities to obtain a cumulative dataset for all entities, which will serve as the data asset inventory result.

[0093] The purpose of step S400 in this embodiment is to achieve accurate screening and comprehensive integration of data assets, ensuring that the final inventory results contain both high-value and highly credible data.

[0094] The logic is as follows: by combining value factors reflecting the uniqueness and usefulness of data with credibility factors reflecting the reliability and normality of data, each piece of secondary data is comprehensively evaluated from the two dimensions of "value" and "credibility," avoiding bias caused by single-dimensional screening. Furthermore, by selecting secondary data that meet the thresholds of the data screening factors, the cumulative dataset is updated and gradually covers all subjects. The final cumulative dataset can completely and accurately represent the data asset status of all subjects, providing a scientific inventory result for the management and application of data assets.

[0095] First, a second set of data filtering factors needs to be calculated based on the second set of data value factors and the second set of data credibility factors. That is, for each piece of second data within the second dataset, a filtering factor is calculated using a preset formula, combining its corresponding second set of data value factors and second set of data credibility factors.

[0096] For example, the preset formula can be a weighted sum: Second data screening factor = Second data value factor × K1 + Second data credibility factor × K2. The weights can be adjusted according to the degree of reliance on data value and credibility. For example, K1=K2=0.5 can be set.

[0097] Integrate all the screening factors of the second data to form a second data screening factor set, and complete the comprehensive quantitative evaluation of each data point.

[0098] For example, if the value factor of a certain second data point of device 15 is 0.8 and the credibility factor is 0.9, then the screening factor of the second data point is 0.8 × 0.5 + 0.9 × 0.5 = 0.85.

[0099] Next, it is necessary to set the data filtering factor threshold. This threshold can be set according to the requirements for the quality of the cumulative dataset, such as 0.6.

[0100] Next, from the second set of data filtering factors, second data with filtering factors greater than or equal to the threshold of the data filtering factor are filtered out and added to the cumulative dataset, so that the cumulative dataset is dynamically updated, which includes high-quality data from the first subject and the second subject.

[0101] For example, if the data filtering factor threshold is 0.6, the second data with a second data filtering factor of 0.85 is retained and added to the cumulative dataset; while the second data with a second data filtering factor of 0.5 is removed.

[0102] Finally, following the above process, the data of all remaining entities, such as the third entity and the fourth entity, are processed in sequence.

[0103] Once all data from all entities has been collected and filtered, the resulting cumulative dataset is the data asset inventory result, which covers high-value and highly credible data from all entities.

[0104] Example 2, as Figure 2 As shown, based on the same inventive concept as the data asset-based inventory method provided in Embodiment 1, this embodiment of the invention also provides a data asset-based inventory system, including:

[0105] The cumulative data acquisition module 11 is used to randomly select a first subject from multiple subjects, collect the first dataset of the first subject as the cumulative dataset, wherein the cumulative dataset is updated as the data of the subjects are collected are traversed;

[0106] Value factor analysis module 12 is used to continue collecting the second dataset of the second subject and analyze the second data value factor set of the second dataset based on the accumulated dataset;

[0107] The data credibility analysis module 13 is used to configure at least one classification data extraction coefficient according to the proportion of data of each subject in the cumulative dataset, extract at least one classification dataset, construct an anomaly classifier, classify multiple second data into anomalies, and obtain a second data credibility factor set.

[0108] The cumulative data filtering module 14 is used to calculate the second data filtering factor set based on the second data value factor set and the second data credibility factor set, filter out data that is greater than or equal to the data filtering factor threshold, add it to the cumulative dataset, and continue filtering to obtain the cumulative dataset of all subjects as the data asset inventory result.

[0109] Furthermore, the cumulative data acquisition module 11 includes the following execution steps:

[0110] Acquire multiple entities whose data assets are to be inventoried;

[0111] Select a first subject among the plurality of subjects, collect a first dataset from the first subject, and use it as a cumulative dataset, wherein the cumulative dataset is updated as the subject data is collected is traversed.

[0112] Furthermore, the value factor analysis module 12 includes the following execution steps:

[0113] Collect a second dataset for the second subject, wherein the second dataset includes multiple pieces of second data;

[0114] Based on the cumulative dataset, analyze the second data value factors of multiple second data points within the second dataset to obtain a set of second data value factors.

[0115] Specifically, based on the cumulative dataset, the second data value factors of multiple second data points within the second dataset are analyzed to obtain a set of second data value factors, including:

[0116] Calculate the similarity between each second data point in the second dataset and all datasets in the cumulative dataset, and calculate the mean to obtain the second data similarity set;

[0117] Based on the second data similarity set, the second data value factor set is calculated.

[0118] Furthermore, the data trust analysis module 13 includes the following execution steps:

[0119] Calculate the proportion of data for each subject within the cumulative dataset, and use it as at least one categorical data extraction coefficient;

[0120] Obtain the preset data extraction quantity, and calculate the data extraction quantity of at least one category by combining the at least one category data extraction coefficient;

[0121] According to the number of data extracted for at least one category, data of at least one subject is randomly extracted from the cumulative dataset to obtain at least one category dataset.

[0122] Using the at least one classification dataset, an anomaly classifier is constructed to classify the second data as anomalies, thereby obtaining a second data confidence factor set.

[0123] Specifically, using at least one classification dataset, an anomaly classifier is constructed to classify multiple pieces of second data as anomalies, thereby obtaining a set of confidence factors for the second data, including:

[0124] Combine the at least one classification dataset to obtain an anomaly classification dataset, randomly select the first anomaly classification data, and construct a layer of classification nodes, wherein the layer of classification nodes performs binary classification on the input data, and labels the data as anomaly data when the data is classified as a single data;

[0125] Continue building to obtain multi-layer classification nodes and obtain an anomaly classifier;

[0126] Multiple pieces of second data are input into the anomaly classifier to obtain the number of node layers at which each piece of second data is classified as an anomaly. The ratio of this number to the maximum number of node layers is calculated to obtain multiple second data credibility factors. These factors are then integrated to obtain a set of second data credibility factors.

[0127] Furthermore, the cumulative data filtering module 14 includes the following execution steps:

[0128] The second data screening factor set is calculated based on the second data value factor set and the second data credibility factor set.

[0129] Filter the data to obtain second data whose second data filtering factor is greater than or equal to the data filtering factor threshold, and add them to the cumulative dataset;

[0130] Continue collecting and filtering data from other entities to obtain a cumulative dataset for all entities, which will serve as the data asset inventory result.

[0131] Example 3, as Figure 3As shown, based on the same inventive concept as the data asset inventory method provided in Embodiment 1, this embodiment of the invention also provides a storage medium 400. For example, the storage medium can be a non-transitory computer-readable storage medium, and the storage medium stores a second computer software program 410. When the second computer software program 410 is executed by a processor, it implements the data asset inventory method as described in Embodiment 1.

[0132] Non-transitory storage media refers to storage media that can still retain data after power is lost, including but not limited to SSDs (solid-state drives), HDDs (hard disk drives), and flash memory devices (USB flash drives, memory cards), etc.

[0133] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0134] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Although preferred embodiments of the invention have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0139] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for inventorying data assets, characterized in that, The method includes: Randomly select a first subject from multiple subjects, collect the first dataset of the first subject as the cumulative dataset, wherein the cumulative dataset is updated as the data of the subjects are collected, the multiple subjects are multiple devices, the first subject is a certain device among the multiple devices, and the first dataset includes the device operation data of the first subject; Continue collecting a second dataset from the second subject. Based on the accumulated dataset, analyze the second data value factor set of the second dataset, including: Collect a second dataset for the second subject, wherein the second dataset includes multiple pieces of second data; Based on the cumulative dataset, the second data value factors of multiple second data points within the second dataset are analyzed to obtain a set of second data value factors, including: Calculate the similarity between each second data point in the second dataset and all datasets in the cumulative dataset, and calculate the mean to obtain the second data similarity set; Based on the second data similarity set, the second data value factor set is calculated and obtained; Based on the proportion of data for each subject in the cumulative dataset, configure at least one classification data extraction coefficient, extract at least one classification dataset, construct an anomaly classifier, classify multiple second data points as anomalies, and obtain a set of second data credibility factors. Based on the second data value factor set and the second data credibility factor set, the second data screening factor set is calculated and obtained. Data with values ​​greater than or equal to the data screening factor threshold is selected and added to the cumulative dataset. The cumulative dataset of all subjects is then selected and used as the data asset inventory result.

2. The inventory method based on data assets according to claim 1, characterized in that, A first subject is randomly selected from multiple subjects, and the first dataset of the first subject is collected as the cumulative dataset, including: Acquire multiple entities whose data assets are to be inventoried; Select a first subject among the plurality of subjects, collect a first dataset from the first subject, and use it as a cumulative dataset, wherein the cumulative dataset is updated as the subject data is collected is traversed.

3. The inventory method based on data assets according to claim 1, characterized in that, Based on the proportion of data for each subject within the accumulated dataset, at least one categorical data extraction coefficient is configured, at least one categorical dataset is extracted, an anomaly classifier is constructed, and multiple pieces of second data are classified as anomalies to obtain a set of second data credibility factors, including: Calculate the proportion of data for each subject within the cumulative dataset, and use it as at least one categorical data extraction coefficient; Obtain the preset data extraction quantity, and calculate the data extraction quantity of at least one category by combining the at least one category data extraction coefficient; According to the number of data extracted for at least one category, data of at least one subject is randomly extracted from the cumulative dataset to obtain at least one category dataset. Using the at least one classification dataset, an anomaly classifier is constructed to classify the second data as anomalies, thereby obtaining a second data confidence factor set.

4. The inventory method based on data assets according to claim 3, characterized in that, Using the at least one classification dataset, an anomaly classifier is constructed to classify multiple pieces of second data as anomalies, thereby obtaining a set of confidence factors for the second data, including: Combine the at least one classification dataset to obtain an anomaly classification dataset, randomly select the first anomaly classification data, and construct a layer of classification nodes, wherein the layer of classification nodes performs binary classification on the input data, and labels the data as anomaly data when the data is classified as a single data; Continue building to obtain multi-layer classification nodes and obtain an anomaly classifier; Multiple pieces of second data are input into the anomaly classifier to obtain the number of node layers at which each piece of second data is classified as an anomaly. The ratio of this number to the maximum number of node layers is calculated to obtain multiple second data credibility factors. These factors are then integrated to obtain a set of second data credibility factors.

5. The inventory method based on data assets according to claim 1, characterized in that, Based on the second data value factor set and the second data credibility factor set, a second data screening factor set is calculated. Data with values ​​greater than or equal to the data screening factor threshold is selected and added to the cumulative dataset. Further screening is then performed to obtain the cumulative dataset for all subjects, which serves as the data asset inventory result, including: The second data screening factor set is calculated based on the second data value factor set and the second data credibility factor set. Filter the data to obtain second data whose second data filtering factor is greater than or equal to the data filtering factor threshold, and add them to the cumulative dataset; Continue collecting and filtering data from other entities to obtain a cumulative dataset for all entities, which will serve as the data asset inventory result.

6. An inventory system based on data assets, characterized in that, The system includes: The cumulative data acquisition module is used to randomly select a first subject from multiple subjects, collect the first dataset of the first subject as the cumulative dataset, wherein the cumulative dataset is updated as the data of the subjects are collected is traversed, the multiple subjects are multiple devices, the first subject is a certain device among the multiple devices, and the first dataset includes the device operation data of the first subject; The value factor analysis module is used to continue collecting a second dataset from the second subject, and based on the accumulated dataset, analyzes the second data value factor set of the second dataset, including: Collect a second dataset for the second subject, wherein the second dataset includes multiple pieces of second data; Based on the cumulative dataset, the second data value factors of multiple second data points within the second dataset are analyzed to obtain a set of second data value factors, including: Calculate the similarity between each second data point in the second dataset and all datasets in the cumulative dataset, and calculate the mean to obtain the second data similarity set; Based on the second data similarity set, the second data value factor set is calculated and obtained; The data credibility analysis module is used to configure at least one classification data extraction coefficient based on the proportion of data of each subject in the cumulative dataset, extract at least one classification dataset, construct an anomaly classifier, classify multiple second data into anomalies, and obtain a second data credibility factor set. The cumulative data filtering module is used to calculate the second data filtering factor set based on the second data value factor set and the second data credibility factor set, filter out data that is greater than or equal to the data filtering factor threshold, add it to the cumulative dataset, and continue filtering to obtain the cumulative dataset of all subjects as the data asset inventory result.

7. A storage medium, characterized in that, The storage medium stores a second computer software program, which, when executed by a processor, implements the data asset-based inventory method as described in any one of claims 1-5.

Images