Data mining method based on data assets

Abstract

The invention discloses a data mining method based on data assets, and relates to the technical field of enterprise management, for the data assets stored in a data platform, the quality characteristics of the data assets are extracted, and the quality grade state of the data assets is judged according to the quality characteristics; when the quality grade state of the data assets stored in the data platform is medium-grade data assets, the data quantity, the data type and the access frequency of the data assets are obtained, and the mining coefficient of the data assets is determined; comparing the minable coefficient of the data assets with a preset minable coefficient threshold, and judging whether to perform data mining on the data assets stored in the data platform or not according to a comparison result; therefore, whether the data assets stored in the data platform can be subjected to data mining or not can be judged, the mining performance of the data assets can be evaluated, it is ensured that the data mining result is meaningful, the calculation cost is reduced, and the possibility that error influences are caused on enterprise management decisions is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise management, and in particular to a data mining method based on data assets. Background Art

[0002] In the field of enterprise management, data mining of enterprise data assets can not only help companies better understand customer needs and optimize business processes, but also improve the accuracy and efficiency of decision-making; for example, retail companies can analyze historical sales data to find out which products perform best in specific seasons, promotions or regions, so as to adjust inventory, pricing and promotion strategies. In addition, data mining can also help companies identify potential risks, such as increased customer churn or supply chain bottlenecks, so as to take countermeasures in advance. For example, a home appliance company analyzed after-sales service data and found that specific products had a high repair rate. Through product improvement or after-sales service optimization, it reduced costs and improved customer satisfaction. These analyses can not only optimize the internal operations of the company, but also provide strong support for strategic decision-making.

[0003] Among them, in the process of data mining on the data assets of an enterprise, generally, the data assets will be stored in the data platform of the enterprise. When data mining is performed, data mining can be performed on the data stored in the data platform;

[0004] However, in the actual data mining process, it is often assumed that the data assets stored in the data platform are mineable. The lack of assessment of the mineability of data assets may lead to meaningless data mining results, blindly increase computing costs, and even cause erroneous impacts on corporate management decisions. Summary of the invention

[0005] The purpose of the present invention is to solve the above-mentioned problems and provide a data mining method based on data assets.

[0006] The present invention provides a data mining method based on data assets, the method comprising:

[0007] For the data assets stored in the data platform, the quality characteristics of the data assets are extracted, and the quality level status of the data assets is determined according to the quality characteristics; the quality level status includes low-level data assets and medium-level data assets;

[0008] When the quality level of the data assets stored in the data platform is a medium-level data asset, the data quantity, data type and access frequency of the data assets are obtained to determine the mining coefficient of the data assets;

[0009] The mining coefficient of the data assets is compared with the preset mining coefficient threshold, and based on the comparison result, it is determined whether to perform data mining on the data assets stored in the data platform.

[0010] Optionally, the quality characteristics of the extracted data assets include a data integrity coefficient, a data uniqueness coefficient, and a data timeliness coefficient of the data assets stored in the data platform.

[0011] Optionally, the calculation steps of the data integrity coefficient and the data uniqueness coefficient are:

[0012] Get the total number of data points in the data asset, get the total number of missing values ​​in the data points, and divide the total number of missing values ​​by the total number of data points to get the data missing ratio; calculate the absolute difference between the data missing ratio and the value 1 as the data integrity coefficient of the data asset;

[0013] Get the total number of data points in the data asset, and get the total number of repetitions of each data point in the data point, calculate the sum of the repetition times of all repeated data points, and get the total number of repetitions of data points in the data asset; divide the total number of repetitions of data points by the total number of data points to get the repetition ratio of the data asset; calculate the absolute difference between the data repetition ratio and the value 1 as the data uniqueness coefficient of the data asset.

[0014] Optionally, the calculation steps of the data timeliness coefficient are:

[0015] For each data point in the data asset, obtain the current timestamp and the last updated timestamp of the data point on the data platform, and calculate the time difference between the current timestamp and the last updated timestamp to obtain the update interval of the corresponding data point; and divide the update interval by the preset maximum update interval to obtain the data timeliness value of the corresponding data point;

[0016] Data points in the data asset whose data timeliness value is greater than 1 are recorded as invalid data points, and the mean of the data timeliness values ​​of the invalid data points is calculated, and the inverse of the mean of the data timeliness values ​​of the invalid data points is taken as the data timeliness coefficient of the data asset.

[0017] Optionally, judging the quality level status of the data asset according to the quality characteristics includes:

[0018] The quality characteristics include data integrity coefficient, data uniqueness coefficient and data timeliness coefficient. The quality assessment coefficient of the data assets stored in the data platform is obtained according to the data integrity coefficient, data uniqueness coefficient and data timeliness coefficient. The calculation formula is: ; In the formula, is the quality assessment coefficient, , , They are data integrity coefficient, data uniqueness coefficient and data timeliness coefficient respectively. They are , , The preset scaling factor of All are greater than 0;

[0019] Compare the quality assessment coefficient of the data assets stored in the data platform with the preset assessment coefficient threshold. If the quality assessment coefficient is less than the preset assessment coefficient threshold, the quality level of the data asset is recorded as a low-level data asset.

[0020] If the quality assessment coefficient is not less than the preset assessment coefficient threshold, the quality level status of the data asset is recorded as a medium-level data asset.

[0021] Optionally, obtaining the data quantity, data type, and access frequency of the data asset and determining the mining coefficient of the data asset includes:

[0022] Obtain the data types of data assets stored in the data platform, obtain the data volume contained in each type, obtain the number of data volumes of this type, and divide the total number of data volumes of this type by the preset minimum data volume to obtain the qualified value of the quantity of data assets of this type;

[0023] Multiply the quantity qualification value of each type of data asset by the preset weight value of the corresponding type of data asset to obtain the data quantity qualification coefficient of the type of data, and perform weighted summation of the mean of the data quantity qualification coefficients of all types of data assets and the number of data types of data assets to obtain the quantity qualification coefficient of data assets;

[0024] The mining coefficient of data assets is determined based on the quantity qualification coefficient, access frequency and quality assessment coefficient of data assets.

[0025] Optionally, determining the mining coefficient of a data asset according to the quantity qualification coefficient, access frequency, and quality assessment coefficient of the data asset includes:

[0026] Obtain the access frequency and access duration of each type of data asset in the data assets stored in the data platform, and use the access frequency and access duration as the data points of the corresponding type of data assets to obtain a data set, and cluster the data set to determine the access coefficient of the data asset; the specific steps are:

[0027] S1: Use the elbow method to determine the optimal number of clusters K for the data set;

[0028] S2: Randomly select K data points in the data set as initial cluster centers. For each data point, calculate its Euclidean distance to each initial cluster center, traverse the K initial cluster centers, and assign it to the cluster corresponding to the nearest initial cluster center.

[0029] S3: After traversing all data points, the initial clusters are obtained. For each initial cluster, the data points in it are averaged to obtain a new cluster center.

[0030] S4: Repeat S2 and S3 until the cluster center no longer changes, and obtain the final cluster and the final cluster center;

[0031] Map the final clusters and the final cluster centers to two-dimensional coordinates, obtain the access coefficients of various types of data assets, and calculate the mean of the access coefficients of all types of data assets as the access coefficient of the data assets;

[0032] The mining coefficient of data assets is determined based on the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets.

[0033] Optionally, the final clusters and the final cluster centers are mapped to two-dimensional coordinates to obtain access coefficients of various types of data assets, including:

[0034] Calculate the weight coefficient of each final cluster, the calculation formula is: ,in Expressed as The weight coefficients of the final clusters, Represented as the number of all data points in the dataset, Indicates The number of data points in the final clusters;

[0035] Calculate the distance of each data point in each final cluster to the coordinate origin on the two-dimensional coordinate to obtain the preliminary coordinate distance of the data point, and multiply the preliminary coordinate distance of the data point by the weight coefficient corresponding to the final cluster in which the data point is located to obtain the actual coordinate distance of the data point;

[0036] The actual coordinate distance of each data point is used as the access coefficient of the corresponding type of data asset.

[0037] Optionally, determining the mining coefficient of the data asset according to the quantity qualification coefficient, access coefficient and quality assessment coefficient of the data asset includes: ; In the formula, is the mining coefficient, , , They are the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets respectively. They are , , The preset scaling factor of Both are greater than 0.

[0038] Optionally, judging whether to perform data mining on the data assets stored in the data platform according to the comparison result includes:

[0039] Compare the mining coefficient of the data asset with the preset mining coefficient threshold. If the mining coefficient is not less than the preset mining coefficient threshold, the data asset stored in the data platform can be mined.

[0040] If the mining coefficient is less than the preset mining coefficient threshold, the data assets stored in the data platform do not need to be mined.

[0041] Beneficial effects of the present invention:

[0042] The present invention proposes a data mining method based on data assets. For data assets stored in a data platform, the quality characteristics of the data assets are extracted, and the quality level status of the data assets is judged according to the quality characteristics; when the data asset status stored in the data platform is a medium-level data asset, the data quantity, data type and access frequency of the data asset are obtained to determine the mineability coefficient of the data asset; the mineability coefficient of the data asset is compared with a preset mineability coefficient threshold, and it is judged whether to perform data mining on the data assets stored in the data platform according to the comparison result; in this way, it is possible to judge whether data assets stored in the data platform can be data mined, evaluate the mineability of the data assets, ensure that the results of data mining are meaningful, reduce computing costs, and reduce the possibility of erroneous impact on enterprise management decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below in conjunction with the accompanying drawings.

[0044] Figure 1 The framework diagram of the data mining method based on data assets. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0047] The embodiment of the present invention provides a data mining method based on data assets. Figure 1 , Figure 1 A framework diagram of a data mining method based on data assets provided by an embodiment of the present invention. The system includes:

[0048] For the data assets stored in the data platform, the quality characteristics of the data assets are extracted, and the quality level status of the data assets is determined based on the quality characteristics; the quality level status includes low-level data assets and medium-level data assets;

[0049] When the quality level of the data assets stored in the data platform is a medium-level data asset, the data quantity, data type and access frequency of the data assets are obtained to determine the mining coefficient of the data assets;

[0050] Compare the mining coefficient of the data asset with the preset mining coefficient threshold, and determine whether to perform data mining on the data assets stored in the data platform based on the comparison result;

[0051] Based on the data asset-based data mining method provided by the embodiment of the present invention, through the above-mentioned method, it is possible to determine whether the data assets stored in the data platform can be mined, evaluate the mineability of the data assets, ensure that the results of data mining are meaningful, reduce computing costs, and reduce the possibility of erroneous impact on enterprise management decisions.

[0052] In one embodiment, the quality characteristics of the extracted data assets include a data integrity coefficient, a data uniqueness coefficient, and a data timeliness coefficient of the data assets stored in the data platform.

[0053] In one embodiment, the steps for calculating the data integrity coefficient and the data uniqueness coefficient are:

[0054] Get the total number of data points in the data asset, get the total number of missing values ​​in the data points, and divide the total number of missing values ​​by the total number of data points to get the data missing ratio; calculate the absolute difference between the data missing ratio and the value 1 as the data integrity coefficient of the data asset;

[0055] Get the total number of data points in the data asset, and get the total number of repetitions of each data point in the data point, calculate the sum of the repetition times of all repeated data points, and get the total number of repetitions of data points in the data asset; divide the total number of repetitions of data points by the total number of data points to get the repetition ratio of the data asset; calculate the absolute difference between the data repetition ratio and the value 1 as the data uniqueness coefficient of the data asset.

[0056] In one embodiment, the steps for calculating the data timeliness coefficient are:

[0057] For each data point in the data asset, obtain the current timestamp and the last updated timestamp of the data point on the data platform, and calculate the time difference between the current timestamp and the last updated timestamp to obtain the update interval of the corresponding data point; and divide the update interval by the preset maximum update interval to obtain the data timeliness value of the corresponding data point;

[0058] Data points in the data asset whose data timeliness value is greater than 1 are recorded as invalid data points, and the mean of the data timeliness values ​​of the invalid data points is calculated, and the inverse of the mean of the data timeliness values ​​of the invalid data points is taken as the data timeliness coefficient of the data asset.

[0059] It should be noted that the data acquisition method involved in the above calculations mainly relies on the log records and metadata management system of the data platform, including: extracting the total number of data points and the number of missing values ​​through the metadata information of the data assets; using data quality detection tools to count the number of duplicate data points; extracting the current time and the last update time of the data point through the timestamp record of the data platform to calculate the update interval; in addition, the preset maximum update interval is usually configured and maintained by the platform administrator based on business needs and historical data characteristics.

[0060] It should be noted that the larger the data integrity coefficient of the data assets stored in the data platform, the higher the quality level of the data assets stored in the enterprise's data platform, because the larger the data integrity coefficient, the fewer missing values ​​in the data points, the stronger the data integrity, and the data assets can provide more accurate and reliable information support in the analysis and mining process. For example, in a retail enterprise's data platform, if there is less missing transaction information in the commodity sales record (such as purchase time, commodity number or customer ID are all fully recorded), the data integrity coefficient of the data set is high, and the enterprise can use this complete data to more accurately predict sales trends, adjust inventory and optimize marketing strategies. If a large number of sales records are missing key fields, the data integrity coefficient will be low, and the reliability of the analysis results and the effectiveness of decision support will be greatly reduced, which may lead to inventory backlogs or loss of sales opportunities. Therefore, a higher data integrity coefficient is directly related to the effective use of data assets and the success rate of enterprise decision-making.

[0061] It should be noted that the larger the data uniqueness coefficient of the data assets stored in the data platform, the higher the quality level of the data assets stored in the enterprise's data platform, because the larger the data uniqueness coefficient, the fewer duplicate values ​​in the data points, the stronger the uniqueness and authenticity of the data, and the more effectively it can support enterprises in data mining and precise analysis. For example, in a customer management system, if the uniqueness coefficient of customer data is high, it means that there are no duplicate records for each customer, such as the customer's name, contact information, address and other information are unique and non-redundant. Such data can help companies accurately calculate the number of customers, analyze customer behavior habits, and design targeted marketing activities. If there are a large number of duplicate entries in the customer records, such as the name and contact information of the same customer being recorded multiple times, the number of customers will be mistakenly overestimated, which will affect the marketing budget and effect evaluation. By maintaining a high data uniqueness coefficient, enterprises can avoid waste of resources and decision-making bias caused by duplicate data, thereby improving the actual value and utilization efficiency of data assets;

[0062] It should be noted that the larger the data timeliness coefficient of the data assets stored in the data platform, the higher the quality level of the data assets stored in the enterprise's data platform, because the larger the data timeliness coefficient, the more timely the data update and the stronger the timeliness, which can provide the most relevant information at the moment and help enterprises make quick responses and decisions. For example, in an e-commerce platform, the update time of commodity inventory is crucial to sales and inventory management. If the data timeliness coefficient is high, it means that the commodity inventory data is updated in time, which can reflect the real inventory situation of the commodity and avoid overselling or inventory backlog. For example, the inventory data of a hot-selling commodity is just updated before the user purchases it. Such data can help the system adjust commodity recommendations, promotional activities and inventory replenishment in real time, thereby improving sales and customer satisfaction. On the contrary, if the inventory data is updated lagging behind, it may cause customers to find that the commodity is out of stock when they purchase it, affecting sales and customer experience. Therefore, a higher data timeliness coefficient ensures that enterprises can make decisions based on the latest data, improve operational efficiency and customer satisfaction.

[0063] In one embodiment, determining the quality level status of a data asset according to the quality characteristics includes:

[0064] The quality characteristics include data integrity coefficient, data uniqueness coefficient and data timeliness coefficient. The quality assessment coefficient of the data assets stored in the data platform is obtained based on the data integrity coefficient, data uniqueness coefficient and data timeliness coefficient. The calculation formula is: ; In the formula, is the quality assessment coefficient, , , They are data integrity coefficient, data uniqueness coefficient and data timeliness coefficient respectively. They are , , The preset scaling factor of All are greater than 0;

[0065] Compare the quality assessment coefficient of the data assets stored in the data platform with the preset assessment coefficient threshold. If the quality assessment coefficient is less than the preset assessment coefficient threshold, the quality level of the data asset is recorded as a low-level data asset.

[0066] If the quality assessment coefficient is not less than the preset assessment coefficient threshold, the quality level status of the data asset is recorded as a medium-level data asset.

[0067] It should be noted that It is set by professionals according to the actual situation. Generally, The sum of is 1, for example They can be 0.3, 0.3, 0.4, or other numbers, without specific limitation. In addition, the preset evaluation coefficient threshold is set by professionals according to actual conditions, and no specific limitation or elaboration is made.

[0068] It should be noted that the reason why the quality level of data assets stored in the data platform is judged from the three aspects of data integrity coefficient, data uniqueness coefficient and data timeliness coefficient is that these three indicators can fully cover the integrity, uniqueness and timeliness of the data, ensuring that the evaluation results reflect the comprehensive quality of the data. The data integrity coefficient can determine whether there are missing or incomplete fields in the data, thereby avoiding information loss in the data analysis and decision-making process; the data uniqueness coefficient can effectively identify whether there is duplicate information in the data, reduce the interference of redundant data on the analysis results, and ensure the accuracy and reliability of the data; the data timeliness coefficient focuses on the frequency of data updates, ensuring that enterprises can make timely decisions based on the latest information and avoid misjudgments caused by outdated data. In contrast, other factors such as data format and storage structure also have a certain impact on data quality, but they cannot directly reflect the applicability and reliability of data. Quality assessment through these three core indicators can more accurately determine whether the data is suitable for mining, helping enterprises avoid waste of resources and decision-making errors.

[0069] In one implementation method, the steps of calculating the data integrity coefficient, data uniqueness coefficient and data timeliness coefficient of the data assets stored in the data platform are relatively simple and uncomplicated, and will not take up too many computing resources. At the same time, by dividing the data assets into low-level and medium-level according to the quality assessment coefficient, it can help enterprises to promptly identify which data are not suitable for in-depth analysis and avoid resource-intensive mining operations when the data quality is poor, thereby reducing computing and analysis costs; in addition, enterprises can use this assessment mechanism to take measures such as supplementation, repair or cleaning for low-level data assets to improve data quality and enhance the effectiveness of future data mining; by comparing with the preset assessment coefficient threshold, it can also regularly track and optimize data quality assessment standards to ensure that data assets always meet business needs and improve the overall credibility and use value of the data platform.

[0070] In one embodiment, when the data asset status stored in the data platform is a medium-level data asset, obtaining the data quantity, data type, and access frequency of the data asset, and determining the mining coefficient of the data asset includes:

[0071] Obtain the data types of data assets stored in the data platform, obtain the data volume contained in each type, obtain the number of data volumes of this type, and divide the total number of data volumes of this type by the preset minimum data volume to obtain the qualified value of the quantity of data assets of this type;

[0072] Multiply the quantity qualification value of each type of data asset by the preset weight value of the corresponding type of data asset to obtain the data quantity qualification coefficient of the type of data, and perform weighted summation of the mean of the data quantity qualification coefficients of all types of data assets and the number of data types of data assets to obtain the quantity qualification coefficient of data assets;

[0073] The mining coefficient of data assets is determined based on the quantity qualification coefficient, access frequency and quality assessment coefficient of data assets.

[0074] It should be noted that the data types of data assets stored in the data platform of an enterprise can include multiple types to ensure that data mining can cover a wider range of fields and needs. For example, structured data, such as sales data, inventory data, financial statements, etc., are usually the most basic data types; unstructured data, such as text data (customer feedback, social media content), image data (product photos, advertising materials) and audio data (customer service call records) can provide support for more complex analysis; time series data, such as website traffic, equipment operation data, etc., can help with trend prediction and anomaly detection; geographic information data (such as location data, map data) can also be used to analyze market areas and user behavior patterns. In addition, combining sensor data (such as temperature and humidity sensor data) and data from Internet of Things (IoT) devices can provide more comprehensive data support for production, transportation and other links, further improving the depth and breadth of data mining; if the number of data types of data assets stored in the data platform of an enterprise is small, it usually means that the data diversity is insufficient and may not be able to support comprehensive analysis and mining. In this case, the enterprise may only rely on some basic data types, such as structured data (such as sales records, financial data) and simple user behavior data (such as click logs, browsing records). Although this type of data can be used for certain specific analyses, such as trend analysis or simple prediction models, the lack of multi-dimensional information sources (such as text data, image data, time series data, etc.) limits the potential for deep data mining. Therefore, the diversity of data types directly affects the complexity and effectiveness of data mining. When there are fewer types of data, the necessity and value of data mining are usually lower.

[0075] It should be noted that the preset minimum data volume and preset weight value are set by professionals based on actual circumstances and are not specifically limited or elaborated.

[0076] It should be noted that the steps of obtaining the quantity qualification coefficient of data assets by weighted summing up the mean of the data quantity qualification coefficients of all types of data assets and the number of data types of data assets are as follows: ; In the formula, is the quantity qualification coefficient, , are the mean value of the data volume qualification coefficient and the number of data types of data assets, respectively. They are , The preset scaling factor of All are greater than 0; It is set by professionals according to the actual situation. Generally, The sum of is 1, for example They can be 0.5, 0.5, or other numbers respectively, and are not specifically limited.

[0077] It should be noted that the greater the quantitative qualification coefficient of data assets, the greater the necessity for data mining of the corresponding enterprise's data assets, because this indicates that the enterprise's data platform stores sufficient data and rich data types, which can provide sufficient data support for in-depth analysis and modeling. For example, if the enterprise stores a large number of data types, including sales data, customer feedback, market analysis reports, and equipment operation data, the amount of data of each data type can provide strong support for different mining goals, thereby helping the enterprise to discover potential market trends, improve operational efficiency, and predict future needs. On the contrary, if the amount of data is insufficient or the type is too single, data mining may not reveal comprehensive insights, resulting in reduced accuracy and reliability of the results. Therefore, when the quantitative qualification coefficient of data assets is large, the necessity and value of data mining also increase.

[0078] In one embodiment, determining the mining coefficient of a data asset according to the quantity qualification coefficient, access frequency, and quality assessment coefficient of the data asset includes:

[0079] Obtain the access frequency and access duration of each type of data asset in the data assets stored in the data platform, and use the access frequency and access duration as the data points of the corresponding type of data assets to obtain a data set, and cluster the data set to determine the access coefficient of the data asset; the specific steps are:

[0080] S1: Use the elbow method to determine the optimal number of clusters K for the data set;

[0081] S2: Randomly select K data points in the data set as initial cluster centers. For each data point, calculate its Euclidean distance to each initial cluster center, traverse the K initial cluster centers, and assign it to the cluster corresponding to the nearest initial cluster center.

[0082] S3: After traversing all data points, the initial clusters are obtained. For each initial cluster, the data points in it are averaged to obtain a new cluster center.

[0083] S4: Repeat S2 and S3 until the cluster center no longer changes, and obtain the final cluster and the final cluster center;

[0084] Map the final clusters and the final cluster centers to two-dimensional coordinates, obtain the access coefficients of various types of data assets, and calculate the mean of the access coefficients of all types of data assets as the access coefficient of the data assets;

[0085] The mining coefficient of data assets is determined based on the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets.

[0086] In one implementation, according to the above method, the optimal clustering results of the data set can be obtained. These clustering results can be used to identify different clusters in the data set, as well as the center and characteristics of each cluster. Such clustering results are helpful for further data analysis and interpretation.

[0087] It should be noted that clustering the data set by K-means clustering is simple and efficient, and can quickly process large-scale data sets; K-means clustering is a commonly used unsupervised learning algorithm, which is used to divide the data set into pre-defined K clusters; the goal of this algorithm is to divide the data points into K clusters, so that each data point belongs to the cluster represented by the cluster center closest to it, and the data points within the cluster are as similar as possible, while the data points between different clusters may not be similar;

[0088] The K-means clustering method classifies these devices into a cluster, and according to the center point of the cluster, an access coefficient can be assigned to each type of data asset; and the access frequency, access duration and access coefficient of each type of data asset show a common linear relationship. The greater the access frequency and the longer the access duration, the greater the access coefficient of the corresponding type of data asset.

[0089] Cluster the data set according to the K-means clustering method:

[0090] Step 1: Use the elbow method to determine the optimal number of clusters K for the data set; the elbow method is one of the common methods used to determine the optimal number of clusters K in K-means clustering. The basic idea is to find an "elbow point" by observing the relationship between the number of clusters K and the total internal deviation square sum (WCSS) of the clustering results. The K value corresponding to the "elbow point" is the position where the WCSS decreases significantly after the number of clusters increases to a certain extent. WCSS refers to the sum of the squares of the distances from each data point to the center of the cluster to which it belongs, which indicates the compactness of the clustering results.

[0091] Step 2: According to the optimal number of clusters K, randomly select K data points from the data points as the initial cluster centers of clustering. The initial cluster centers refer to the K cluster centers randomly selected before starting the clustering process. These initial cluster centers are the starting points when the algorithm starts.

[0092] Step 3: For each data point in the data set, calculate the distance between it and each initial cluster center using the Euclidean distance. For example, calculate the data point and the initial cluster centers The distance between , and its calculation formula is ,in, Represents data points In the The values ​​in the clusters, Represents the initial cluster center In the The value of the clusters, Represented as data points and the initial cluster centers The number of features is calculated, and the data points are assigned to the cluster to which the nearest cluster center belongs. Euclidean distance refers to the distance between two points in Euclidean space, and is also one of the most commonly used distance metrics. In K-means clustering, Euclidean distance is usually used to calculate the distance between data points and cluster centers.

[0093] Step 4: After traversing all data points, we get the initial clusters. For each initial cluster, we calculate the mean of the data points in it. The calculation formula is: ,in Expressed as The set of data points in a cluster, Expressed as The new cluster centers of the clusters;

[0094] Step 5: Repeat steps 3 and 4 until the cluster center no longer changes, and obtain the final cluster and the final cluster center.

[0095] It should be specifically explained that the step of assigning a data point to the cluster to which the nearest cluster center belongs is to sort the distances between the data point and each cluster center from large to small, and select the smallest distance to assign the data point to the cluster to which it belongs.

[0096] In one embodiment, the final clusters and the final cluster centers are mapped to two-dimensional coordinates, and the access coefficients of various types of data assets are obtained, including:

[0097] Calculate the weight coefficient of each final cluster, the calculation formula is: ,in Expressed as The weight coefficients of the final clusters, Represented as the number of all data points in the dataset, Indicates The number of data points in the final clusters;

[0098] Calculate the distance of each data point in each final cluster to the coordinate origin on the two-dimensional coordinate to obtain the preliminary coordinate distance of the data point, and multiply the preliminary coordinate distance of the data point by the weight coefficient corresponding to the final cluster in which the data point is located to obtain the actual coordinate distance of the data point;

[0099] The actual coordinate distance of each data point is used as the access coefficient of the corresponding type of data asset.

[0100] In one implementation, through cluster analysis and weighted distance calculation, the access levels of all types of data assets in the data assets can be comprehensively considered.

[0101] It should be noted that, because the access frequency and access duration are selected as data points for clustering, these two data points can now be used as coordinates and mapped to the two-dimensional coordinate axis. The access coefficient of each type of data asset will be mapped to a point on the two-dimensional coordinate axis. After the clustering is completed, all data points will be distributed in different final clusters on the two-dimensional coordinate axis. The closer the final cluster is to the original center and the data points it includes, the larger the access coefficient of the corresponding type of data assets, and vice versa. At the same time, the weight coefficient of each final cluster is calculated to correct the distance from the data point to the original center. In this way, the different importance of each final cluster can be ensured, and the impact of noise and outliers on the clustering results can be reduced to a certain extent, so that the access coefficient of the final calculated data asset is more accurate.

[0102] It should be noted that when the frequency of access and the duration of access to data assets are greater, the greater the access coefficient, the greater the necessity for data mining of the corresponding enterprise's data assets, because when the frequency of access and the duration of access to data assets are greater, it means that these data assets have received more attention and use on the platform, reflecting their importance and value in actual business. Frequent access usually indicates that the data has a high dependence on business processes, decisions or customer experience, while a longer access duration may indicate that users or systems have conducted in-depth analysis and use of the data. This high frequency and long-term access indicates that the quality and practicality of the data assets are higher and more in line with the conditions for data mining. For enterprises, this type of data asset can provide more insights and value, so the necessity for its mining also increases; for example, if the product data of an e-commerce platform is frequently accessed by a large number of users and each visit lasts for a long time, this indicates that the data is crucial to the platform's sales, recommendation system and other functions, and is therefore more suitable for in-depth analysis and data mining; therefore, the greater the access coefficient, the greater the mining potential of the data asset.

[0103] In one embodiment, determining the mining coefficient of a data asset according to the quantity qualification coefficient, access coefficient, and quality assessment coefficient of the data asset includes: ; In the formula, is the mining coefficient, , , They are the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets respectively. They are , , The preset scaling factor of All are greater than 0;

[0104] It should be noted that It is set by professionals according to the actual situation. Generally, The sum of is 1, for example They can be 024, 0.43, 0.33 or other numbers respectively, and are not limited to any specific number.

[0105] In one embodiment, comparing the mining coefficient of the data asset with a preset mining coefficient threshold, and determining whether to perform data mining on the data asset stored in the data platform according to the comparison result includes:

[0106] Compare the mining coefficient of the data asset with the preset mining coefficient threshold. If the mining coefficient is not less than the preset mining coefficient threshold, the data asset stored in the data platform can be mined.

[0107] If the mining coefficient is less than the preset mining coefficient threshold, the data assets stored in the data platform do not need to be mined.

[0108] It should be noted that the preset mining coefficient threshold is set by professionals according to actual conditions and will not be limited or elaborated on in detail.

[0109] In one implementation, the mining coefficient of the data asset is calculated. This coefficient combines indicators such as the quantity qualification coefficient and the access coefficient of the data, and can reflect the quality, availability and activity of the data on the platform; a preset mining coefficient threshold is set as the standard for data mining. If the mining coefficient of the data asset is greater than or equal to the preset threshold, it means that the data asset meets the conditions for data mining and has sufficient potential to provide in-depth analysis support for the business, so data mining can be carried out; if the mining coefficient is less than the preset threshold, it means that the value and utilization rate of the data asset are low, and the cost of data mining may be greater than the benefit, so mining is not recommended. The advantage of doing so is that it can help enterprises accurately evaluate which data is worth investing resources for in-depth analysis, thereby avoiding ineffective waste of resources. By setting clear thresholds, enterprises can focus on mining data with high value and potential, while ensuring the effectiveness and cost-effectiveness of data mining. For example, on an e-commerce platform, product data that is frequently accessed and has rich historical data can be mined first for accurate recommendation and user behavior analysis, while for inventory data that is less visited and updated less frequently, data mining can be postponed or abandoned to avoid wasting resources. Such a strategy helps to optimize resource allocation and improve the efficiency of data analysis and the quality of decision-making.

[0110] It should be noted that the data assets stored in the enterprise's data platform can be mined in a variety of ways, and the specific method to be selected should be determined according to the business needs, data characteristics and management goals of the enterprise. First of all, enterprises should classify and manage data according to the type and quality of data, and adopt different mining methods for different data assets. For example, for customer behavior data, cluster analysis and association rule mining techniques can be used to gain insight into customers' purchasing preferences and consumption habits. Based on this, enterprises can carry out precision marketing to improve customer satisfaction and conversion rates. For product performance data, regression analysis and predictive modeling can be used to evaluate the future trends and potential risks of products, helping enterprises to optimize products and manage risks. In addition, enterprises can also use natural language processing (NLP) technology to analyze large amounts of text data to reveal customers' emotional tendencies and brand awareness, thereby optimizing marketing strategies.

[0111] At the enterprise management level, data mining is not only a technical operation of data, but also a support tool for strategic decision-making. Enterprise managers can gain in-depth insights through data mining and promote intelligent decision-making processes. For example, sales forecasts based on data mining can help sales teams rationally plan sales targets and inventory management, avoid excessive backlogs or out-of-stock phenomena, and thus improve overall operational efficiency and profits. In terms of supply chain management, data mining can help companies identify bottlenecks and potential risks in the supply chain, make timely adjustments, and improve the resilience and flexibility of the supply chain. In human resource management, by analyzing employee performance, turnover rate, satisfaction and other data, companies can predict the risk of employee turnover, formulate corresponding talent retention plans and incentive policies, and ensure the stability and efficiency of the team.

[0112] At the same time, the effective implementation of data mining requires enterprises to have strong data management capabilities and technical support, including management measures in data cleaning, data integration, data security, etc. Enterprises need to ensure the controllability and accuracy of data quality by establishing a sound data governance system. The management of the enterprise should regularly review the results of data mining, evaluate its actual contribution to the business, and continuously optimize data mining strategies and technical applications according to market changes and internal demand adjustments to ensure the continued value growth of data assets. Through such comprehensive management, data mining can not only improve the operational efficiency of enterprises, but also help enterprises maintain competitive advantages in a complex and changing market environment and achieve sustainable development.

[0113] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be used to artificially limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A data mining method based on data assets, characterized in that: The following steps are involved: For the data assets stored in the data platform, the quality characteristics of the data assets are extracted, and the quality level status of the data assets is determined according to the quality characteristics; the quality level status includes low-level data assets and medium-level data assets; When the quality level of the data assets stored in the data platform is a medium-level data asset, the data quantity, data type and access frequency of the data assets are obtained to determine the mining coefficient of the data assets; The mining coefficient of the data assets is compared with the preset mining coefficient threshold, and based on the comparison result, it is determined whether to perform data mining on the data assets stored in the data platform.

2. The data mining method based on data assets according to claim 1, characterized in that: The quality characteristics of the extracted data assets include the data integrity coefficient, data uniqueness coefficient and data timeliness coefficient of the data assets stored in the data platform.

3. The data mining method based on data assets according to claim 2, characterized in that: The calculation steps of data integrity coefficient and data unique coefficient are as follows: Get the total number of data points in the data asset, and the total number of missing values ​​in the data points, and divide the total number of missing values ​​by the total number of data points to get the data missing ratio; Calculate the absolute difference between the missing data ratio and the value 1 as the data integrity coefficient of the data asset; Obtain the total number of data points in the data asset, obtain the total number of repetitions of each data point in the data point, calculate the sum of the number of repetitions of all repeated data points, and obtain the total number of repetitions of data points in the data asset; divide the total number of repetitions of data points by the total number of data points to obtain the repetition ratio of the data asset; The absolute difference between the data duplication ratio and the value 1 is calculated as the data uniqueness coefficient of the data asset.

4. The data mining method based on data assets according to claim 2, characterized in that: The calculation steps of data timeliness coefficient are as follows: For each data point in the data asset, obtain the current timestamp and the last updated timestamp of the data point on the data platform, and calculate the time difference between the current timestamp and the last updated timestamp to obtain the update interval of the corresponding data point; And divide the update interval by the preset maximum update interval to obtain the data timeliness value of the corresponding data point; Data points in the data asset whose data timeliness value is greater than 1 are recorded as invalid data points, and the mean of the data timeliness values ​​of the invalid data points is calculated, and the inverse of the mean of the data timeliness values ​​of the invalid data points is taken as the data timeliness coefficient of the data asset.

5. The data mining method based on data assets according to claim 1, characterized in that: Judging the quality level of data assets based on quality characteristics includes: The quality characteristics include data integrity coefficient, data uniqueness coefficient and data timeliness coefficient. The quality assessment coefficient of the data assets stored in the data platform is obtained according to the data integrity coefficient, data uniqueness coefficient and data timeliness coefficient. The calculation formula is: ; In the formula, is the quality assessment coefficient, , , They are data integrity coefficient, data uniqueness coefficient and data timeliness coefficient respectively. They are , , The preset scaling factor of All are greater than 0; Compare the quality assessment coefficient of the data assets stored in the data platform with the preset assessment coefficient threshold. If the quality assessment coefficient is less than the preset assessment coefficient threshold, the quality level of the data asset is recorded as a low-level data asset. If the quality assessment coefficient is not less than the preset assessment coefficient threshold, the quality level status of the data asset is recorded as a medium-level data asset.

6. The data mining method based on data assets according to claim 5, characterized in that: Obtaining the data quantity, data type, and access frequency of data assets and determining the mining coefficient of data assets include: Obtain the data types of data assets stored in the data platform, obtain the data volume contained in each type, obtain the number of data volumes of this type, and divide the total number of data volumes of this type by the preset minimum data volume to obtain the qualified value of the quantity of data assets of this type; Multiply the quantity qualification value of each type of data asset by the preset weight value of the corresponding type of data asset to obtain the data quantity qualification coefficient of the type of data, and perform weighted summation of the mean of the data quantity qualification coefficients of all types of data assets and the number of data types of data assets to obtain the quantity qualification coefficient of data assets; The mining coefficient of data assets is determined based on the quantity qualification coefficient, access frequency and quality assessment coefficient of data assets.

7. The data mining method based on data assets according to claim 6, characterized in that: The mining coefficient of data assets is determined based on the quantity qualification coefficient, access frequency and quality assessment coefficient of data assets, including: Obtain the access frequency and access duration of each type of data asset in the data assets stored in the data platform, and use the access frequency and access duration as the data points of the corresponding type of data assets to obtain a data set, and cluster the data set to determine the access coefficient of the data asset; the specific steps are: S1: Use the elbow method to determine the optimal number of clusters K for the data set; S2: Randomly select K data points in the data set as initial cluster centers. For each data point, calculate its Euclidean distance to each initial cluster center, traverse the K initial cluster centers, and assign it to the cluster corresponding to the nearest initial cluster center. S3: After traversing all data points, the initial clusters are obtained. For each initial cluster, the data points in it are averaged to obtain a new cluster center. S4: Repeat S2 and S3 until the cluster center no longer changes, and obtain the final cluster and the final cluster center; Map the final clusters and the final cluster centers to two-dimensional coordinates, obtain the access coefficients of various types of data assets, and calculate the mean of the access coefficients of all types of data assets as the access coefficient of the data assets; The mining coefficient of data assets is determined based on the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets.

8. The data mining method based on data assets according to claim 7, characterized in that: Mapping the final clusters and the final cluster centers to two-dimensional coordinates, the access coefficients of various types of data assets are obtained: Calculate the weight coefficient of each final cluster, the calculation formula is: ,in Expressed as The weight coefficients of the final clusters, Represented as the number of all data points in the dataset, Indicates The number of data points in the final clusters; Calculate the distance of each data point in each final cluster to the coordinate origin on the two-dimensional coordinate to obtain the preliminary coordinate distance of the data point, and multiply the preliminary coordinate distance of the data point by the weight coefficient corresponding to the final cluster in which the data point is located to obtain the actual coordinate distance of the data point; The actual coordinate distance of each data point is used as the access coefficient of the corresponding type of data asset.

9. The data mining method based on data assets according to claim 7, characterized in that: The mining coefficient of data assets is determined based on the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets, including: ; In the formula, is the mining coefficient, , , They are the quantity qualification coefficient, access coefficient and quality assessment coefficient of data assets respectively. They are , , The preset scaling factor of Both are greater than 0.

10. The data mining method based on data assets according to claim 1, characterized in that: Judging whether to conduct data mining on the data assets stored in the data platform based on the comparison results includes: Compare the mining coefficient of the data asset with the preset mining coefficient threshold. If the mining coefficient is not less than the preset mining coefficient threshold, the data asset stored in the data platform can be mined. If the mining coefficient is less than the preset mining coefficient threshold, the data assets stored in the data platform do not need to be mined.

Images