Data management method and system based on knowledge graph

Abstract

The present invention discloses a data management method and system based on a knowledge graph, including: obtaining historical record data of various fields and standardizing it to obtain standardized data corresponding to each field; constructing a knowledge graph of the corresponding field based on the standardized data corresponding to each field to obtain a standard knowledge graph library; obtaining data to be processed and determining the field information of the data to be processed; querying the standard knowledge graph library based on the field information of the data to be processed to determine a target knowledge graph of the field corresponding to the data to be processed; managing the data to be processed based on the target knowledge graph; constructing knowledge graphs for different fields, and determining the target knowledge graph based on the field of the data to be processed, so as to more accurately classify and manage the data to be processed and improve the efficiency and accuracy of data management.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a data management method and system based on a knowledge graph. Background Art

[0002] With the advent of the big data era, data management is becoming more and more important in life. Various data play an important role in the business management process. With the rapid development of digitalization, there is more and more data, and the amount of data is getting bigger and bigger. The integration of data from different fields has resulted in lower and lower efficiency in data management, and management has become more and more difficult. Knowledge graphs are widely used in the field of data management. Therefore, it is currently necessary to propose a method based on knowledge graphs to achieve rapid management of data, thereby improving data management efficiency. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, the first aspect of the present invention aims to propose a data management method based on a knowledge graph, construct knowledge graphs for different fields, determine the target knowledge graph based on the field of the data to be processed, and classify and manage the data to be processed more accurately, thereby improving the efficiency and accuracy of data management.

[0004] The second aspect of the present invention aims to propose a data management system based on knowledge graph.

[0005] To achieve the above objectives, the first embodiment of the present invention proposes a data management method based on a knowledge graph, comprising:

[0006] Obtain historical record data in various fields and standardize them to obtain standardized data corresponding to each field;

[0007] Based on the standardized data corresponding to each field, a knowledge graph of the corresponding field is constructed to obtain a standard knowledge graph library;

[0008] Obtain the data to be processed and determine the domain information of the data to be processed;

[0009] Based on the domain information of the data to be processed, the standard knowledge graph database is queried to determine the target knowledge graph corresponding to the domain of the data to be processed;

[0010] Data management is performed on the data to be processed based on the target knowledge graph.

[0011] Preferably, historical record data in various fields are obtained and standardized to obtain standardized data corresponding to each field, including:

[0012] Perform data standard conversion processing on historical record data in various fields to obtain initial standard data; the data standard conversion processing includes data encoding rules, data attribute rules and data structure rules;

[0013] The initial standard data is subjected to data quality control processing to obtain standardized data corresponding to each field; the data quality control processing includes data uniqueness and data integrity.

[0014] Preferably, constructing a knowledge graph for a corresponding field includes:

[0015] Select standardized data corresponding to a field as the sample data set;

[0016] Extract entity information and association relationships from sample data sets;

[0017] Determine the node attributes of the knowledge graph according to the entity information;

[0018] Determine the node relationship of the knowledge graph according to the association relationship;

[0019] Based on the node attributes and the node relationships, a knowledge graph of the corresponding field is constructed.

[0020] Preferably, before performing data management on the data to be processed based on the target knowledge graph, the method further includes:

[0021] Obtain a time series sequence corresponding to the data to be processed, and divide the time series sequence into a plurality of time series subsequences; obtain the data length corresponding to each time series subsequence;

[0022] Randomly select a time series subsequence as a target time series subsequence; calculate the absolute value of the difference between the data length of the target time series subsequence and all time series subsequences except the target time series subsequence, and obtain a plurality of absolute values ​​of the difference; and obtain the maximum value and the average value of the plurality of absolute values ​​of the difference;

[0023] The ratio of the mean value to the maximum value is used as the data importance value corresponding to the target time series subsequence;

[0024] Traverse all time series subsequences to obtain the data importance value corresponding to each time series subsequence;

[0025] Compare the data importance value corresponding to the time series subsequence with the data importance threshold; take the subsequence data corresponding to the time series subsequence whose data importance value is greater than or equal to the data importance threshold as the first data to be cleaned; take the subsequence data corresponding to the time series subsequence whose data importance value is less than the data importance threshold as the second data to be cleaned;

[0026] Cleaning the first data to be cleaned based on a first cleaning rule to obtain first cleaned data; the first cleaning rule is a combination of multiple data cleaning rules;

[0027] Inputting the second data to be cleaned into a pre-trained data anomaly detection model to determine the proportions of various anomaly types corresponding to the second data to be cleaned;

[0028] Obtain the data cleaning rule corresponding to the maximum value of the abnormal type ratio as the second cleaning rule;

[0029] Performing data cleaning on the second data to be cleaned based on the second cleaning rule to obtain second cleaned data;

[0030] The cleaned data to be processed are obtained according to the first cleaned data and the second cleaned data.

[0031] Preferably, performing data cleaning on the first data to be cleaned based on the first cleaning rule includes:

[0032] Inputting the first data to be cleaned into a pre-trained data classification model to obtain a plurality of classified data;

[0033] Obtain any classification data as target classification data; calculate the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data, and obtain several similarity values ​​corresponding to each target classification sub-data;

[0034] Calculate the sum of several similarity values ​​to obtain the category center evaluation value corresponding to each target classification sub-data; take the target classification sub-data corresponding to the maximum category center evaluation value as the category center of the target classification data;

[0035] Calculate the Euclidean distance between the data center point of each group of target classification sub-data and the data center point of the target classification sub-data corresponding to the category center;

[0036] Comparing the Euclidean distance with a preset distance threshold, and when it is determined that the Euclidean distance is greater than or equal to the preset distance threshold, using the target classification sub-data as target cleaning data;

[0037] Data cleaning is performed on target cleaning data included in the first data to be cleaned based on the first cleaning rule.

[0038] Preferably, the calculating the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data includes:

[0039]

[0040] in, Represents the first The target classification sub-data and the target classification data except The target classification sub-data The similarity between target classification sub-data; Indicates The total number of target feature vectors in the target classification sub-data; Indicates The target classification sub-data The value of the target feature vector; Indicates The target classification sub-data The value corresponding to the eigenvector; Indicates The total number of feature vectors in the target classification sub-data; =1, 2, 3, ... N; =1, 2, 3…M.

[0041] Preferably, after performing data cleaning on the target cleaning data based on the first cleaning rule, the method further includes:

[0042] Recalculate the Euclidean distance between the data center point corresponding to the target cleaned data after data cleaning and the data center point of the target classified sub-data corresponding to the category center;

[0043] When it is determined that the Euclidean distance is less than the preset distance threshold, data cleaning of the target cleaning data is completed.

[0044] Preferably, the determining of the domain information of the data to be processed includes:

[0045] The data to be processed is input into the pre-trained data domain classification model for classification to determine the domain information corresponding to the data to be processed.

[0046] Preferably, the data management for the data to be processed based on the target knowledge graph includes:

[0047] Input the data to be processed into a pre-trained data classification model to obtain a number of sub-data to be processed, and obtain the data label corresponding to each sub-data to be processed;

[0048] Query the target knowledge graph based on the data label corresponding to each sub-data to be processed, and determine the position of each sub-data to be processed in the target knowledge graph;

[0049] Data management is performed on the data to be processed based on the position of each sub-data to be processed in the target knowledge graph.

[0050] To achieve the above objectives, the second aspect of the present invention proposes a data management system based on a knowledge graph, including:

[0051] The first acquisition module is used to acquire historical record data in various fields and perform standardized processing to obtain standardized data corresponding to each field;

[0052] Establish a module for constructing a knowledge graph in the corresponding field based on the standardized data corresponding to each field, and obtain a standard knowledge graph library;

[0053] The second acquisition module is used to acquire the data to be processed and determine the domain information of the data to be processed;

[0054] A determination module is used to query the standard knowledge graph library based on the domain information of the data to be processed, and determine the target knowledge graph corresponding to the domain of the data to be processed;

[0055] A management module is used to perform data management on the data to be processed based on the target knowledge graph.

[0056] The present invention discloses a data management method based on knowledge graph. By acquiring historical record data in various fields and performing standardization processing, it can ensure that data in different fields have a unified format and structure, which facilitates subsequent data management and analysis; standardization processing also helps to eliminate noise and errors in data and improve data quality; knowledge graph is a powerful knowledge representation and reasoning tool that can reveal the complex relationships between data; constructing a knowledge graph of the corresponding field based on standardized data can integrate scattered data into an organic whole, which is convenient for users to understand and query; the formation of a standard knowledge graph library enables knowledge graphs in different fields to be interconnected to form a cross-domain knowledge network; by determining the field information of the data to be processed and automatically querying the standard knowledge graph library to find the corresponding target knowledge graph, the The method can realize automatic classification and identification of data; data management based on the target knowledge graph can realize functions such as intelligent recommendation, associated query, and anomaly detection of data, thereby improving the efficiency and accuracy of data management; since the knowledge graph is modular and extensible, the method can easily add new domain data and knowledge graphs to adapt to changing business needs; at the same time, the update and maintenance of the knowledge graph are also relatively simple, and only the knowledge graph of a specific field needs to be modified, which will not affect the operation of the entire system; the data management method based on the knowledge graph has significant advantages and beneficial effects in data standardization, knowledge representation, automated management, extensibility, decision support, and user-friendliness; the present invention also discloses a data management system based on the knowledge graph, and the specific beneficial effects are not elaborated here.

[0057] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0058] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 is a flow chart of a data management method based on a knowledge graph according to an embodiment of the present invention;

[0061] Figure 2 is a flowchart of constructing a knowledge graph of a corresponding field according to an embodiment of the present invention;

[0062] Figure 3 It is a block diagram of a data management system based on a knowledge graph according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0064] Example 1

[0065] like Figure 1 As shown, the first embodiment of the present invention provides a data management method based on a knowledge graph, including steps S1-S5:

[0066] S1: Obtain historical record data in various fields and standardize them to obtain standardized data corresponding to each field;

[0067] S2: Build the knowledge graph of the corresponding field based on the standardized data corresponding to each field to obtain the standard knowledge graph library;

[0068] S3: Obtain the data to be processed and determine the domain information of the data to be processed;

[0069] S4: query the standard knowledge graph library based on the domain information of the data to be processed, and determine the target knowledge graph corresponding to the domain of the data to be processed;

[0070] S5: Perform data management on the data to be processed based on the target knowledge graph.

[0071] In this embodiment, by acquiring historical record data in various fields and performing standardization processing, it can be ensured that data in different fields have a unified format and structure, which facilitates subsequent data management and analysis; standardization processing also helps to eliminate noise and errors in the data and improve data quality.

[0072] In this embodiment, the standardization process includes data standard conversion process and data quality control process.

[0073] In this embodiment, before data management is performed on the data to be processed based on the target knowledge graph, the data to be processed is also preprocessed, including data cleaning, noise reduction, etc.

[0074] The beneficial effects of the above technical solution are: by obtaining historical record data in various fields and performing standardized processing, it can be ensured that data in different fields have a unified format and structure, which facilitates subsequent data management and analysis; standardized processing also helps to eliminate noise and errors in the data and improve data quality; knowledge graph is a powerful knowledge representation and reasoning tool that can reveal the complex relationships between data; constructing a knowledge graph of the corresponding field based on standardized data can integrate scattered data into an organic whole, which is convenient for users to understand and query; the formation of a standard knowledge graph library enables knowledge graphs in different fields to be interconnected to form a cross-domain knowledge network; by determining the field information of the data to be processed and automatically querying the standard knowledge graph library In order to find the corresponding target knowledge graph, this method can realize automatic classification and identification of data; data management based on the target knowledge graph can realize functions such as intelligent data recommendation, associated query, anomaly detection, etc., and improve the efficiency and accuracy of data management; since the knowledge graph is modular and extensible, this method can easily add new domain data and knowledge graphs to adapt to changing business needs; at the same time, the update and maintenance of the knowledge graph is relatively simple, and only the knowledge graph of a specific field needs to be modified, which will not affect the operation of the entire system; the data management method based on the knowledge graph has significant advantages and beneficial effects in data standardization, knowledge representation, automated management, scalability, decision support and user-friendliness.

[0075] Example 2

[0076] Obtain historical record data in various fields and standardize them to obtain standardized data corresponding to each field, including:

[0077] Perform data standard conversion processing on historical record data in various fields to obtain initial standard data; the data standard conversion processing includes data encoding rules, data attribute rules and data structure rules;

[0078] The initial standard data is subjected to data quality control processing to obtain standardized data corresponding to each field; the data quality control processing includes data uniqueness and data integrity.

[0079] In this embodiment, the data encoding rule refers to the way of encoding and decoding data, which ensures that different systems or applications can correctly understand and interpret data; the encoding rule may include character encoding, digital encoding, encoding of specific identifiers, etc.

[0080] In this embodiment, the data attribute rules define the characteristics and properties of the data; this includes data type, field length, required fields, data format, etc.; the purpose of the attribute rules is to ensure the consistency and accuracy of the data.

[0081] In this embodiment, the data structure rule describes the organization and relationship of data; it includes the structure of the data table, the association between fields, the definition of primary keys and foreign keys, etc.; the data structure rule helps to ensure the integrity and consistency of the data.

[0082] In this embodiment, the data uniqueness and integrity processed by data quality control may include: data verification: verifying the input data to ensure that it complies with the specified data format, range and constraints; duplicate data detection: taking appropriate measures to remove or process duplicate data by comparing and identifying duplicate data elements; data consistency check: checking the consistency of data between different tables, files or systems to ensure the consistency and accuracy of the data; data repair: repairing and supplementing any data errors or missing values ​​found to ensure the integrity of the data.

[0083] The beneficial effects of the above technical solution are: the historical record data is converted to data standards to ensure that the data follows unified coding rules, attribute rules and structural rules; this helps to improve the standardization and consistency of data, so that data from different fields can be better compared and integrated with each other; by performing data quality control on the initial standard data, including checking the uniqueness and integrity of the data, errors, duplications and missing values ​​in the data can be discovered and resolved. This helps to improve the quality of the data and reduce the impact of data errors on subsequent analysis and decision-making; after obtaining standardized data, different users and systems can use and understand the data more conveniently. Standardized data is also easier to integrate and share with other data sources, improving the availability and reliability of data; by standardizing the historical record data in various fields, cross-domain data analysis and comparison can be better performed. This helps to discover the connections and trends between different fields and provide more comprehensive information for comprehensive decision-making; standardized data processing and data quality control can help organizations establish effective data management processes and specifications and improve the efficiency and accuracy of data management.

[0084] Example 3

[0085] like Figure 2 As shown, a knowledge graph of the corresponding field is constructed, including S21-S25:

[0086] S21: Select standardized data corresponding to a field as a sample data set;

[0087] S22: extracting entity information and association relationships in the sample data set;

[0088] S23: Determine the node attributes of the knowledge graph according to the entity information;

[0089] S24: Determine the node relationship of the knowledge graph according to the association relationship;

[0090] S25: Based on the node attributes and the node relationships, construct a knowledge graph of the corresponding field.

[0091] The beneficial effects of the above technical solution are: by constructing a knowledge graph, the entity information and association relationships within the field can be visualized, which is conducive to a clearer understanding and grasp of the relationship between the various parts, thereby improving the efficiency of knowledge organization and management; the knowledge graph can show the connection between entities and relationships, which is helpful to discover potential new knowledge and relationships, and provide valuable inspiration for further research and exploration; the association relationships in the knowledge graph can help solve complex problems, such as inferring the connection between problems through association relationships, and then formulating more effective solutions; building a knowledge graph is conducive to the sharing and dissemination of knowledge, and facilitates experts and researchers in different fields to better use existing knowledge and promote the development of the field.

[0092] Example 4

[0093] Before performing data management on the data to be processed based on the target knowledge graph, the method further includes:

[0094] Obtain a time series sequence corresponding to the data to be processed, and divide the time series sequence into a plurality of time series subsequences; obtain the data length corresponding to each time series subsequence;

[0095] Randomly select a time series subsequence as a target time series subsequence; calculate the absolute value of the difference between the data length of the target time series subsequence and all time series subsequences except the target time series subsequence, and obtain a plurality of absolute values ​​of the difference; and obtain the maximum value and the average value of the plurality of absolute values ​​of the difference;

[0096] The ratio of the mean value to the maximum value is used as the data importance value corresponding to the target time series subsequence;

[0097] Traverse all time series subsequences to obtain the data importance value corresponding to each time series subsequence;

[0098] Compare the data importance value corresponding to the time series subsequence with the data importance threshold; take the subsequence data corresponding to the time series subsequence whose data importance value is greater than or equal to the data importance threshold as the first data to be cleaned; take the subsequence data corresponding to the time series subsequence whose data importance value is less than the data importance threshold as the second data to be cleaned;

[0099] Cleaning the first data to be cleaned based on a first cleaning rule to obtain first cleaned data; the first cleaning rule is a combination of multiple data cleaning rules;

[0100] Inputting the second data to be cleaned into a pre-trained data anomaly detection model to determine the proportions of various anomaly types corresponding to the second data to be cleaned;

[0101] Obtain the data cleaning rule corresponding to the maximum value of the abnormal type ratio as the second cleaning rule;

[0102] Performing data cleaning on the second data to be cleaned based on the second cleaning rule to obtain second cleaned data;

[0103] The cleaned data to be processed are obtained according to the first cleaned data and the second cleaned data.

[0104] In this embodiment, the time series segmentation method may be segmentation based on fixed time intervals.

[0105] In this embodiment, it is assumed that the time series corresponding to the data to be processed is divided into 5 time series subsequences A, B, C, D, and E, and the corresponding data lengths are 30, 50, 47, 65, and 82; when A is used as the target time series subsequence, the absolute value of the difference between the data lengths of A and B, C, D, and E is calculated to obtain , , , At this time, the maximum absolute value of the difference is 52, and the mean is (20+17+35+52) / 4=31. At this time, the data importance value corresponding to the target time series sequence A is 31 / 52.

[0106] In this embodiment, the second data to be cleaned is input into a pre-trained data anomaly detection model to determine the proportion of various anomaly types corresponding to the second data to be cleaned. Since the second data to be cleaned is screened out data with relatively low importance, this type of data will not have much impact on subsequent data applications. Therefore, in order to improve the efficiency of data cleaning, the data cleaning rule corresponding to the maximum value of the anomaly type proportion in this type of data is obtained to perform data cleaning.

[0107] The beneficial effects of the above technical solution are: by segmenting the time series corresponding to the data to be processed, calculating the importance value of each time series, and comparing the importance value of each time series with the preset data importance threshold, the data to be processed is classified, so that data cleaning can be performed more targeted and the efficiency and quality of data cleaning can be improved; for the first data to be cleaned with higher data importance, a combination of multiple data cleaning rules is used for cleaning, which can more comprehensively deal with possible problems; for the second data to be cleaned with lower data importance, the corresponding cleaning rules are selected according to the proportion of abnormal types, saving cleaning costs; the comprehensive use of different cleaning rules can better adapt to different types of data and cleaning needs; the data to be processed after cleaning is finally obtained with higher quality and is more conducive to subsequent data management and analysis.

[0108] Example 5

[0109] Performing data cleaning on the first data to be cleaned based on the first cleaning rule includes:

[0110] Inputting the first data to be cleaned into a pre-trained data classification model to obtain a plurality of classified data;

[0111] Obtain any classification data as target classification data; calculate the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data, and obtain several similarity values ​​corresponding to each target classification sub-data;

[0112] Calculate the sum of several similarity values ​​to obtain the category center evaluation value corresponding to each target classification sub-data; take the target classification sub-data corresponding to the maximum category center evaluation value as the category center of the target classification data;

[0113] Calculate the Euclidean distance between the data center point of each group of target classification sub-data and the data center point of the target classification sub-data corresponding to the category center;

[0114] Comparing the Euclidean distance with a preset distance threshold, and when it is determined that the Euclidean distance is greater than or equal to the preset distance threshold, using the target classification sub-data as target cleaning data;

[0115] Data cleaning is performed on target cleaning data included in the first data to be cleaned based on the first cleaning rule.

[0116] The beneficial effects of the above technical solution are: through classification and similarity calculation, the data that needs to be cleaned can be identified more accurately; only the part determined as the target cleaning data is cleaned, reducing unnecessary processing steps; based on the comparison of data center points and Euclidean distances, the distribution characteristics of the data are taken into account to make cleaning more targeted; according to specific business needs and data characteristics, parameters such as preset distance thresholds can be adjusted to adapt to different cleaning requirements; through classification and center calculation, outlier or abnormal data can be effectively removed to improve the quality of data cleaning.

[0117] Example 6

[0118] The calculating the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data includes:

[0119]

[0120] in, Represents the first The target classification sub-data and the target classification data except The target classification sub-data The similarity between target classification sub-data; Indicates The total number of target feature vectors in the target classification sub-data; Indicates The target classification sub-data The value of the target feature vector; Indicates The target classification sub-data The value corresponding to the eigenvector; Indicates The total number of feature vectors in the target classification sub-data; =1, 2, 3, ... N; =1, 2, 3…M.

[0121] The beneficial effects of the above technical solution are: it takes into account the values ​​of multiple eigenvectors in the target classification sub-data, which can more comprehensively reflect the similarity between data; it counts the number of eigenvectors, making the similarity calculation more accurate; it covers multiple dimensions of data, which helps to discover potential similar patterns; it can flexibly adjust the weights and number of eigenvectors according to specific needs to improve the flexibility of calculation; and it provides a reliable similarity measurement basis for subsequent classification and cleaning operations.

[0122] Example 7

[0123] After the target cleaning data is cleaned based on the first cleaning rule, the method further includes:

[0124] Recalculate the Euclidean distance between the data center point corresponding to the target cleaned data after data cleaning and the data center point of the target classified sub-data corresponding to the category center;

[0125] When it is determined that the Euclidean distance is less than the preset distance threshold, data cleaning of the target cleaning data is completed.

[0126] The beneficial effects of the above technical solution are: by recalculating the Euclidean distance and comparing it with the preset threshold, the cleaning effect is further verified to ensure that the target cleaned data is closer to the category center; improve data quality: make the cleaned data more consistent and neat; reduce miscleaning: avoid excessive or insufficient cleaning to ensure the rationality of the data; optimize data distribution: promote data to move closer to the category center and improve data availability; provide a better basis for subsequent analysis: the cleaned data is more suitable for further analysis and processing.

[0127] Example 8

[0128] The determining of the domain information of the data to be processed includes:

[0129] The data to be processed is input into the pre-trained data domain classification model for classification to determine the domain information corresponding to the data to be processed.

[0130] In this embodiment, the pre-trained data domain classification model is a data domain classification model trained in advance.

[0131] The beneficial effects of the above technical solution are: by using the pre-trained data domain classification model, the domain information of the data to be processed can be quickly and accurately determined; automatic classification is achieved, manual intervention is reduced, and processing efficiency is improved; it ensures that the data is accurately classified into the corresponding field, providing more targeted support for subsequent processing; it can process various types and formats of data to be processed, and has good versatility.

[0132] Example 9

[0133] The data management of the data to be processed based on the target knowledge graph includes:

[0134] Input the data to be processed into a pre-trained data classification model to obtain a number of sub-data to be processed, and obtain the data label corresponding to each sub-data to be processed;

[0135] Query the target knowledge graph based on the data label corresponding to each sub-data to be processed, and determine the position of each sub-data to be processed in the target knowledge graph;

[0136] Data management is performed on the data to be processed based on the position of each sub-data to be processed in the target knowledge graph.

[0137] In this embodiment, according to the position of each sub-data to be processed in the target knowledge graph, it also includes: storing each sub-data to be processed in a corresponding database; setting a unique identifier for each sub-data to be processed so as to track and identify it in data management and subsequent processing; establishing an index to quickly retrieve and query the sub-data to be processed at a specific location; regularly backing up data to prevent data loss or damage; setting appropriate access rights to ensure that only authorized personnel can access and operate data at a specific location; updating data in a timely manner as needed to ensure the accuracy and completeness of the data; integrating the sub-data to be processed with other related data to provide a more comprehensive data analysis perspective; and using encryption, access control and other security measures to protect data security.

[0138] The beneficial effects of the above technical solution are: by subdividing the data to be processed and determining its position in the knowledge graph, the data can be managed and organized more accurately; based on the data labels and the position of the knowledge graph, the classification and storage of data can be ensured to be more accurate; according to the position of the sub-data in the knowledge graph, the required data can be queried and retrieved more quickly; it helps to optimize subsequent data processing steps, making the processing more efficient and orderly; and provides a more targeted and accurate data foundation for data analysis.

[0139] like Figure 3 As shown, the second embodiment of the present invention provides a data management system based on a knowledge graph, including:

[0140] The first acquisition module is used to acquire historical record data in various fields and perform standardized processing to obtain standardized data corresponding to each field;

[0141] Establish a module for constructing a knowledge graph in the corresponding field based on the standardized data corresponding to each field, and obtain a standard knowledge graph library;

[0142] The second acquisition module is used to acquire the data to be processed and determine the domain information of the data to be processed;

[0143] A determination module is used to query the standard knowledge graph library based on the domain information of the data to be processed, and determine the target knowledge graph corresponding to the domain of the data to be processed;

[0144] A management module is used to perform data management on the data to be processed based on the target knowledge graph.

[0145] The beneficial effects of the above technical solution are: by obtaining historical record data in various fields and performing standardized processing, it can be ensured that data in different fields have a unified format and structure, which facilitates subsequent data management and analysis; standardized processing also helps to eliminate noise and errors in the data and improve data quality; knowledge graph is a powerful knowledge representation and reasoning tool that can reveal the complex relationships between data; constructing a knowledge graph of the corresponding field based on standardized data can integrate scattered data into an organic whole, which is convenient for users to understand and query; the formation of a standard knowledge graph library enables knowledge graphs in different fields to be interconnected to form a cross-domain knowledge network; by determining the field information of the data to be processed and automatically querying the standard knowledge graph library In order to find the corresponding target knowledge graph, this method can realize automatic classification and identification of data; data management based on the target knowledge graph can realize functions such as intelligent data recommendation, associated query, anomaly detection, etc., and improve the efficiency and accuracy of data management; since the knowledge graph is modular and extensible, this method can easily add new domain data and knowledge graphs to adapt to changing business needs; at the same time, the update and maintenance of the knowledge graph is relatively simple, and only the knowledge graph of a specific field needs to be modified, which will not affect the operation of the entire system; the data management method based on the knowledge graph has significant advantages and beneficial effects in data standardization, knowledge representation, automated management, scalability, decision support and user-friendliness.

Claims

1. A data management method based on knowledge graph, characterized in that: include: Obtain historical record data in various fields and standardize them to obtain standardized data corresponding to each field; Based on the standardized data corresponding to each field, a knowledge graph of the corresponding field is constructed to obtain a standard knowledge graph library; Obtain the data to be processed and determine the domain information of the data to be processed; Based on the domain information of the data to be processed, the standard knowledge graph database is queried to determine the target knowledge graph corresponding to the domain of the data to be processed; Performing data management on the data to be processed based on the target knowledge graph; Before performing data management on the data to be processed based on the target knowledge graph, the method further includes: Obtain a time series sequence corresponding to the data to be processed, and divide the time series sequence into a plurality of time series subsequences; obtain the data length corresponding to each time series subsequence; Randomly select a time series subsequence as a target time series subsequence; calculate the absolute value of the difference between the data length of the target time series subsequence and all time series subsequences except the target time series subsequence, and obtain a plurality of absolute values ​​of the difference; and obtain the maximum value and the average value of the plurality of absolute values ​​of the difference; The ratio of the mean value to the maximum value is used as the data importance value corresponding to the target time series subsequence; Traverse all time series subsequences to obtain the data importance value corresponding to each time series subsequence; Compare the data importance value corresponding to the time series subsequence with the data importance threshold; take the subsequence data corresponding to the time series subsequence whose data importance value is greater than or equal to the data importance threshold as the first data to be cleaned; take the subsequence data corresponding to the time series subsequence whose data importance value is less than the data importance threshold as the second data to be cleaned; Cleaning the first data to be cleaned based on a first cleaning rule to obtain first cleaned data; the first cleaning rule is a combination of multiple data cleaning rules; Inputting the second data to be cleaned into a pre-trained data anomaly detection model to determine the ratio of various abnormal data types corresponding to the second data to be cleaned; Obtaining a data cleaning rule corresponding to the abnormal type with the maximum value in the data type ratio as the second cleaning rule; Performing data cleaning on the second data to be cleaned based on the second cleaning rule to obtain second cleaned data; Obtaining cleaned data to be processed according to the first cleaned data and the second cleaned data; Performing data cleaning on the first data to be cleaned based on the first cleaning rule includes: Inputting the first data to be cleaned into a pre-trained data classification model to obtain a plurality of classified data; Obtain any classification data as target classification data; calculate the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data, and obtain several similarity values ​​corresponding to each target classification sub-data; Calculate the sum of several similarity values ​​to obtain the category center evaluation value corresponding to each target classification sub-data; take the target classification sub-data corresponding to the maximum category center evaluation value as the category center of the target classification data; Calculate the Euclidean distance between the data center point of each group of target classification sub-data and the data center point of the target classification sub-data corresponding to the category center; Comparing the Euclidean distance with a preset distance threshold, and when it is determined that the Euclidean distance is greater than or equal to the preset distance threshold, using the target classification sub-data as target cleaning data; Data cleaning is performed on target cleaning data included in the first data to be cleaned based on the first cleaning rule.

2. The data management method based on knowledge graph according to claim 1, characterized in that: Obtain historical record data in various fields and standardize them to obtain standardized data corresponding to each field, including: Perform data standard conversion processing on historical record data in various fields to obtain initial standard data; the data standard conversion processing includes data encoding rules, data attribute rules and data structure rules; The initial standard data is subjected to data quality control processing to obtain standardized data corresponding to each field; the data quality control processing includes data uniqueness and data integrity.

3. The data management method based on knowledge graph according to claim 1, characterized in that: Construct the knowledge graph of the corresponding field, including: Select standardized data corresponding to a field as the sample data set; Extract entity information and association relationships from sample data sets; Determine the node attributes of the knowledge graph according to the entity information; Determine the node relationship of the knowledge graph according to the association relationship; Based on the node attributes and the node relationships, a knowledge graph of the corresponding field is constructed.

4. The data management method based on knowledge graph according to claim 1, characterized in that: The calculating the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data includes: Among them, d i，j represents the similarity between the i-th target classification sub-data in the target classification data and the j-th target classification sub-data in the target classification data except the i-th target classification sub-data; N represents the total number of target feature vectors in the i-th target classification sub-data; T i，n represents the value of the nth target feature vector in the i-th target classification sub-data; P j，m represents the value corresponding to the mth eigenvector in the jth target classification sub-data; M represents the total number of eigenvectors in the jth target classification sub-data; n=1, 2, 3...N; m=1, 2, 3...M.

5. The data management method based on knowledge graph according to claim 1, characterized in that: After the target cleaning data is cleaned based on the first cleaning rule, the method further includes: Recalculate the Euclidean distance between the data center point corresponding to the target cleaned data after data cleaning and the data center point of the target classified sub-data corresponding to the category center; When it is determined that the Euclidean distance is less than the preset distance threshold, data cleaning of the target cleaning data is completed.

6. The data management method based on knowledge graph according to claim 1, characterized in that: The determining of the domain information of the data to be processed includes: The data to be processed is input into the pre-trained data domain classification model for classification to determine the domain information corresponding to the data to be processed.

7. The data management method based on knowledge graph according to claim 1, characterized in that: The data management of the data to be processed based on the target knowledge graph includes: Input the data to be processed into a pre-trained data classification model to obtain a number of sub-data to be processed, and obtain the data label corresponding to each sub-data to be processed; Query the target knowledge graph based on the data label corresponding to each sub-data to be processed, and determine the position of each sub-data to be processed in the target knowledge graph; Data management is performed on the data to be processed based on the position of each sub-data to be processed in the target knowledge graph.

8. A data management system based on knowledge graph, characterized in that: include: The first acquisition module is used to acquire historical record data in various fields and perform standardized processing to obtain standardized data corresponding to each field; Establish a module for constructing a knowledge graph in the corresponding field based on the standardized data corresponding to each field, and obtain a standard knowledge graph library; The second acquisition module is used to acquire the data to be processed and determine the domain information of the data to be processed; A determination module is used to query the standard knowledge graph library based on the domain information of the data to be processed, and determine the target knowledge graph corresponding to the domain of the data to be processed; A management module, used for performing data management on the data to be processed based on the target knowledge graph; Before the management module manages the data to be processed based on the target knowledge graph, it also includes: Obtain a time series sequence corresponding to the data to be processed, and divide the time series sequence into a plurality of time series subsequences; obtain the data length corresponding to each time series subsequence; Randomly select a time series subsequence as a target time series subsequence; calculate the absolute value of the difference between the data length of the target time series subsequence and all time series subsequences except the target time series subsequence, and obtain a plurality of absolute values ​​of the difference; and obtain the maximum value and the average value of the plurality of absolute values ​​of the difference; The ratio of the mean value to the maximum value is used as the data importance value corresponding to the target time series subsequence; Traverse all time series subsequences to obtain the data importance value corresponding to each time series subsequence; Compare the data importance value corresponding to the time series subsequence with the data importance threshold; take the subsequence data corresponding to the time series subsequence whose data importance value is greater than or equal to the data importance threshold as the first data to be cleaned; take the subsequence data corresponding to the time series subsequence whose data importance value is less than the data importance threshold as the second data to be cleaned; Cleaning the first data to be cleaned based on a first cleaning rule to obtain first cleaned data; the first cleaning rule is a combination of multiple data cleaning rules; Inputting the second data to be cleaned into a pre-trained data anomaly detection model to determine the ratio of various abnormal data types corresponding to the second data to be cleaned; Obtaining a data cleaning rule corresponding to the abnormal type with the maximum value in the data type ratio as the second cleaning rule; Performing data cleaning on the second data to be cleaned based on the second cleaning rule to obtain second cleaned data; Obtaining cleaned data to be processed according to the first cleaned data and the second cleaned data; Performing data cleaning on the first data to be cleaned based on the first cleaning rule includes: Inputting the first data to be cleaned into a pre-trained data classification model to obtain a plurality of classified data; Obtain any classification data as target classification data; calculate the similarity between each target classification sub-data in the target classification data and other target classification sub-data in the target classification data, and obtain several similarity values ​​corresponding to each target classification sub-data; Calculate the sum of several similarity values ​​to obtain the category center evaluation value corresponding to each target classification sub-data; take the target classification sub-data corresponding to the maximum category center evaluation value as the category center of the target classification data; Calculate the Euclidean distance between the data center point of each group of target classification sub-data and the data center point of the target classification sub-data corresponding to the category center; Comparing the Euclidean distance with a preset distance threshold, and when it is determined that the Euclidean distance is greater than or equal to the preset distance threshold, using the target classification sub-data as target cleaning data; Data cleaning is performed on target cleaning data included in the first data to be cleaned based on the first cleaning rule.

Images