Intelligent data management system based on large model

Abstract

The invention discloses an intelligent data governance system based on a large model, and relates to the technical field of data analysis, the intelligent data governance system comprises a data acquisition module, a data cleaning fusion module, a model training module, a deviation trend analysis module, a data annotation analysis module and a data correction governance module, the modules are in electric signal connection; the data acquisition module is used for acquiring basic data from a data source for implementing data governance, and the basic data comprises historical data and current data. According to the method, the deviation condition in the data is automatically identified and labeled by integrating the data labeling analysis module, accurate positioning is provided for subsequent data correction, the most suitable correction method is intelligently matched and applied in combination with the correction measure library, efficient data management is achieved, in addition, the data change is monitored, the deviation value before and after correction is compared, and the accuracy of data processing is improved. And the effectiveness of correction measures is ensured, so that the overall quality and accuracy of the treated data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to an intelligent data management system based on a large model. Background Art

[0002] With the continuous advancement of big data technology, the amount of data faced by entities such as enterprises and social organizations is growing explosively. The data comes from various channels including social media, IoT devices, enterprise business systems, etc., and is massive, high-speed, and diverse. Therefore, it is necessary to conduct intelligent data governance on such a large amount of data to meet the growth and high-speed requirements of data volume. The development of big model technology provides key technical support for intelligent data governance. Big models have powerful semantic understanding and reasoning capabilities, which can automatically identify and correct errors in data and improve the efficiency and accuracy of data cleaning.

[0003] Since there are data deviations in the process of data acquisition, which affects the accuracy and precision of the data application process, how to identify the degree of deviation in the data and implement corresponding correction and governance measures in a targeted manner is the problem we need to solve. To this end, an intelligent data governance system based on a large model is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent data governance system based on a large model to solve the problems raised in the above-mentioned background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: An intelligent data management system based on a large model, the intelligent data management system includes a data acquisition module, a data cleaning and fusion module, a model training module, a deviation trend analysis module, a data annotation analysis module and a data correction management module, wherein the modules are connected by electrical signals; The data acquisition module is used to acquire basic data from the data source for implementing data governance, wherein the basic data includes historical data and current data, to ensure efficient aggregation and consistency of data, and to provide a basis for subsequent data processing and analysis; The data cleaning and fusion module pre-processes the collected basic data and integrates the data to obtain a data fusion sequence, thereby enhancing the integrity and consistency of the data and providing high-quality data support for subsequent data analysis and model training; The model training module trains a deviation analysis model based on historical data in the data fusion sequence to analyze the deviation of the data; The deviation trend analysis module determines the baseline data for data governance and calculates the data deviation value to clarify the degree of deviation by combining the analysis results of the data deviation situation with historical data; The data annotation analysis module presets a deviation threshold based on the deviation degree analysis result, determines the deviation degree of the deviation data, and performs data annotation; The data correction and management module clarifies the degree of deviation of the current data and matches corresponding correction measures for data management.

[0006] A further improvement of the technical solution of the present invention is that in the data acquisition module, the basic data acquisition process is: According to the goals and needs of data governance, identify the types of data sources that need to be connected to implement data governance, including internal business systems, external partner systems, social media platforms, and IoT devices, and establish connections with the corresponding data sources; Define data extraction rules according to business needs and data governance strategies, determine the data fields, data formats, and data frequencies to be extracted, and set the priority and sequence of data extraction to ensure priority processing of key data. According to the defined rules, extract basic data including historical data and current data from the connected data sources; For historical data, batch collection is used to obtain the required data stored in the database at one time. For current data, real-time collection is used to continuously obtain data through real-time data streams. The collected data is converted into a unified format to facilitate subsequent data processing and analysis, and the data is standardized to eliminate differences between different data sources and ensure data consistency.

[0007] A further improvement of the technical solution of the present invention is that in the data cleaning and fusion module, the integration and acquisition process of the data fusion sequence is as follows: Import the collected basic data into the data cleaning and fusion module, and conduct preliminary analysis on the data to clarify the structure, type, distribution and potential problems of the data; Scan the data set integrated with basic data and perform data cleaning, including missing value processing, outlier detection and duplicate data processing. According to the characteristics of the data and business needs, identify the fields and records where the missing values ​​are located, and implement processing measures to fill in or delete the missing values; Horizontally integrate data records of the same entity from different data sources, and integrate data records of different time periods in the same data source to form a continuous time series data set; Create a unique ID number for each set of data, and establish an association between each data and the ID number based on the timestamp association attribute between the data, and integrate them to form a data fusion sequence.

[0008] A further improvement of the technical solution of the present invention is that in the model training module, the analysis process of the data deviation is as follows: The historical data part is extracted from the data fusion sequence as the input of the training deviation analysis model, and the support vector machine model is selected to train the deviation analysis model. The historical data part contains the relevant information and features of each entity in the past period of time, which can reflect the changing rules and trends of the data in different time periods; Perform feature analysis on historical data, extract features for deviation analysis, including standard deviation, skewness and kurtosis, and integrate various features to obtain feature data sets. The standard deviation is used to measure the degree of dispersion of data distribution, indicating the average deviation between data points and the mean. The larger the standard deviation, the greater the volatility of the data, indicating that the difference between data points is large. The standard deviation can be used to evaluate the stability and consistency of data and help identify abnormal fluctuations in data. Skewness is used to measure the asymmetry of data distribution and indicates the direction and degree of skewness of data distribution. A positive skewness indicates that the data is right-skewed, that is, the right tail of the data is longer, and a negative skewness indicates that the data is left-skewed, that is, the left tail of the data is longer. Skewness can help identify whether the shape of data distribution deviates from normal distribution, thereby discovering potential deviations. Kurtosis is used to measure the sharpness of data distribution and indicates the shape of the peak of data distribution. Kurtosis can be used to evaluate the extreme value of data distribution and help identify outliers in data. Each sample in the feature data set corresponds to an entity in historical data, including the standard deviation, skewness and kurtosis characteristic values ​​of the entity, as well as the corresponding deviation label. The deviation label indicates whether there is a deviation. Divide the feature data set into a training set and a test set. Use the training set data combined with the support vector machine model to train the deviation analysis model. Optimize the model performance by adjusting the model parameters. Use the cross-validation method to evaluate the model's stability and generalization ability. Use the test set to evaluate the trained model. Calculate the model's accuracy, recall, and F1 score evaluation indicators. Based on the evaluation results, determine whether the model's performance meets business requirements. Use the trained model to perform deviation analysis on the data in the data fusion sequence, identify the deviation in the data, predict whether the data has deviation, explain the identified deviation, and analyze the cause and impact of the deviation.

[0009] A further improvement of the technical solution of the present invention is that in the deviation trend analysis module, the calculation process of the data deviation value is: Traverse the analysis results and historical data containing data deviation in the data fusion sequence to obtain relevant information and features of each entity in different time periods; Combine historical data with business needs, select a fixed evaluation period and data range from historical data, and determine benchmark data including standard deviation, skewness, and kurtosis characteristics as the benchmark data for data governance. The benchmark data is the average value of each characteristic in the historical data. Compare and analyze the differences between each feature in each data point and its corresponding benchmark data one by one, and calculate the feature deviation value; Integrate the deviation values ​​of each feature, calculate the data deviation value comprehensively, analyze the calculated data deviation value, and clarify the deviation degree and distribution of the data; By analyzing the changing trend of deviation values ​​over time, the deviation trend of the data can be identified, and chart tools (line charts) can be used to display the changes in deviation values ​​over time. The key nodes and turning points in the deviation trend can be observed, and the causes and impacts of the deviation can be analyzed.

[0010] A further improvement of the technical solution of the present invention is that the calculation expression of the data deviation value is: ; In the formula, is the data deviation value, which is used to measure the degree of data deviation. is the standard deviation eigenvalue, which indicates the degree of dispersion of the data. is the benchmark data of the standard deviation, that is, the average value of the standard deviation in the historical data. is the skewness eigenvalue, which indicates the asymmetry of the data distribution. is the benchmark data of skewness, that is, the average value of skewness in historical data, is the kurtosis eigenvalue, which indicates the sharpness of the data distribution. is the benchmark data of kurtosis, that is, the average value of kurtosis in historical data, , , They represent the minimum values ​​of the standard deviation, skewness, and kurtosis eigenvalues, respectively, and are used for normalization to ensure that the terms in the formula are comparable. The value range is between 0 and 1. When it is close to 0, it means that the deviation between the data and the benchmark data is very small and the data is relatively stable.

[0011] A further improvement of the technical solution of the present invention is that in the data annotation analysis module, the data annotation process is: Obtain the calculated data deviation value from the deviation trend analysis module, use the data deviation value to analyze the difference between each data point and the benchmark data, and measure the data deviation; Based on business needs and data deviation values, different deviation thresholds are preset to determine deviation data, and deviation data is divided into different deviation levels, namely slight deviation, medium deviation and severe deviation; The calculated data deviation value is compared with the preset deviation threshold, and the deviation level of the data point is marked according to the comparison result, and the marked data is then stored in different data sets for subsequent processing.

[0012] A further improvement of the technical solution of the present invention is that: a plurality of the deviation levels are provided with a plurality of the deviation thresholds in one-to-one correspondence, specifically satisfying the following relationship: Slight deviations: ; Medium deviation: ; Serious deviations: ; in, is the data deviation value, is the upper threshold corresponding to slight deviation and the lower threshold corresponding to medium deviation, The upper threshold for moderate deviations and the lower threshold for severe deviations.

[0013] A further improvement of the technical solution of the present invention is that in the data correction management module, the process of data management is: Obtain annotated deviation data from the data annotation analysis module, including the characteristics of the data point, the degree of deviation, and the degree of deviation, and clarify the degree of deviation of each data point based on the annotation information of the deviation data; Based on different phenomena of deviation data in historical data, a correction measure library containing predefined solutions for different deviation degrees and types is built to implement data governance for deviation data; According to the assessed degree of deviation, the corresponding correction measures are matched from the correction measures library, and the selected correction measures are applied to the deviation data. The correction measures include data resampling, data interpolation, data smoothing, data conversion and correction of data processing logic, etc. For deviations caused by data collection errors, the data are recollected to obtain more accurate data information. For deviations caused by missing values ​​or outliers, interpolation methods are used to fill or correct the data. For deviations caused by large data fluctuations, smoothing algorithms are used to smooth the data to reduce the impact of short-term fluctuations. For deviations caused by uneven data distribution or skewness, data conversion is performed to make the data distribution more uniform and symmetrical. For deviations caused by improper data processing, the logic and algorithm of data processing are corrected to ensure the correctness and consistency of data processing; After correction, monitor the data changes and compare the data deviation values ​​before and after correction to evaluate the effect of data governance. If the governance effect is not ideal, re-analyze the cause of the deviation based on the verification results, adjust the correction measures, and perform the correction operation again until a satisfactory governance effect is achieved; Store the governed data in the data warehouse, replace the original deviation data, and record the detailed information of the data governance activities, including corrective measures, data changes before and after correction, and correction time.

[0014] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art: 1. The present invention provides an intelligent data governance system based on a large model. By integrating a data annotation and analysis module, it automatically identifies and annotates deviations in the data, provides precise positioning for subsequent data correction, and combines with a correction measure library to intelligently match and apply the most suitable correction method to achieve efficient data governance. In addition, it monitors data changes, compares deviation values ​​before and after correction, ensures the effectiveness of correction measures, and thus improves the overall quality and accuracy of the data after governance.

[0015] 2. The present invention provides an intelligent data governance system based on a large model. The deviation analysis model is trained through the model training module to accurately analyze the deviation of the data. The deviation value of the data is calculated and the degree of deviation of the data is analyzed in combination with the deviation threshold to ensure that the corresponding correction measures are matched and the data governance process is efficiently implemented to further improve the quality of the data. The corrected data is stored in the data warehouse to provide a high-quality data foundation for subsequent operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0017] Figure 1 It is a module diagram of the present invention; Figure 2 The figure is a flow chart for calculating the data deviation value of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Embodiment 1, as Figure 1 As shown, the present invention provides an intelligent data management system based on a large model, the intelligent data management system includes a data acquisition module, a data cleaning and fusion module, a model training module, a deviation trend analysis module, a data annotation analysis module and a data correction management module, wherein the modules are connected by electrical signals; The data acquisition module is used to obtain basic data from the data source for data governance, where the basic data includes historical data and current data, to ensure efficient data aggregation and consistency, and to provide a basis for subsequent data processing and analysis. According to the goals and needs of data governance, the data source types that need to be connected to implement data governance are identified, including internal business systems, external partner systems, social media platforms, and IoT devices, and connections are established with the corresponding data sources. The ways to establish connections include API interface docking, database connection configuration, and network protocol adaptation. For data sources that provide API interfaces, to obtain data by calling the API interface, it is necessary to negotiate with the data source to obtain the API key, understand the API usage restrictions and data format, etc., to ensure that the required data can be obtained smoothly through the API interface. For data sources stored in the database, configure the database connection parameters, including database type, server address, port number, user name and Password, etc., to establish a connection with the database in order to extract data from the database. For IoT devices that transmit data over the network, adapt the corresponding network protocol to ensure that the device can communicate normally with the data acquisition module and transmit data in real time. According to business needs and data governance strategies, define data extraction rules, determine the data fields, data formats and data frequencies to be extracted, and formulate the priority and sequence of data extraction to ensure priority processing of key data. According to the defined rules, extract basic data including historical data and current data from the data source to which the connection is established. For historical data, use batch collection to obtain the required data stored in the database at one time. For current data, use real-time collection to continuously obtain data through real-time data streams. Convert the collected data into a unified format to facilitate subsequent data processing and analysis, and standardize the data to eliminate differences between different data sources to ensure data consistency. The data cleaning and fusion module pre-processes the collected basic data and integrates the data to obtain a data fusion sequence to enhance the integrity and consistency of the data, providing high-quality data support for subsequent data analysis and model training. The collected basic data are uniformly imported into the data cleaning and fusion module, and a preliminary analysis of the data is performed to clarify the structure, type, distribution and potential problems of the data. The data set that integrates the basic data is scanned for data cleaning, including missing value processing, outlier detection and duplicate data processing. According to the characteristics of the data and business needs, the fields and records where the missing values ​​are located are identified, and the processing measures of filling in or deleting the missing values ​​are implemented. For numerical fields, the mean, median or mode of the field is used to fill in the missing values. For categorical fields, the mode is used to fill in. For data with obvious trends or periodicity, interpolation filling is implemented. For records or fields with many missing values ​​that cannot be accurately filled, choose to delete them to avoid a significant impact on subsequent analysis and model training. For detected outliers, take corresponding processing measures according to the nature of the outliers and business needs, including correcting outliers, deleting outliers, and retaining outliers. Use data deduplication algorithms to compare the values ​​of each field of data records and identify duplicate records. After identifying duplicate data, delete redundant duplicate records according to business needs and data uniqueness requirements, and only retain one valid data. Horizontally integrate data records of the same entity from different data sources, and integrate data records from different time periods in the same data source to form a continuous time series data set. Create a unique ID number for each set of data, and establish an association relationship between each data and the ID number based on the timestamp association attribute between the data, and integrate to form a data fusion sequence; The model training module trains the deviation analysis model based on the historical data in the data fusion sequence, analyzes the deviation of the data, extracts the historical data part from the data fusion sequence as the input of the training deviation analysis model, and selects the support vector machine model to train the deviation analysis model. The historical data part contains the relevant information and features of each entity in the past period of time, which can reflect the changing rules and trends of the data in different time periods. The historical data is subjected to feature analysis, and the features used for deviation analysis are extracted, including standard deviation, skewness and kurtosis. The features are integrated to obtain the feature data set. Among them, the standard deviation is used to measure the degree of dispersion of the data distribution, which indicates the average deviation between the data point and the mean. The larger the standard deviation, the greater the volatility of the data, indicating that the difference between the data points is large. The standard deviation can be used to evaluate the stability and consistency of the data and help identify abnormal fluctuations in the data. The skewness is used to measure the asymmetry of the data distribution, which indicates the skew direction and degree of the data distribution. A positive skewness indicates that the data is right-skewed, that is, the right tail of the data is longer, and a negative skewness indicates that the data is left-skewed, that is, the left tail of the data is longer. The skewness can help identify the data. Whether the shape of the data distribution deviates from the normal distribution, so as to find potential deviations. Kurtosis is used to measure the sharpness of the data distribution and represents the shape of the peak of the data distribution. Kurtosis can be used to evaluate the extreme value of the data distribution and help identify outliers in the data. Each sample in the feature data set corresponds to an entity in the historical data, including the standard deviation, skewness and kurtosis characteristic values ​​of the entity, as well as the corresponding deviation label. The deviation label is whether there is a deviation. The feature data set is divided into a training set and a test set. The deviation analysis model is trained using the training set data combined with the support vector machine model. The performance of the model is optimized by adjusting the model parameters. The stability and generalization ability of the model are evaluated using the cross-validation method. The trained model is evaluated using the test set. The accuracy, recall and F1 score evaluation indicators of the model are calculated. According to the evaluation results, it is judged whether the performance of the model meets the business needs. The trained model is used to perform deviation analysis on the data in the data fusion sequence, identify the deviation in the data, predict whether the data has deviation, and explain the identified deviation, and analyze the causes and impacts of the deviation. The deviation trend analysis module combines the analysis results of data deviation and historical data to determine the baseline data for data governance and calculate the data deviation value to clarify the degree of deviation; The data annotation analysis module presets the deviation threshold based on the deviation degree analysis results, determines the deviation degree of the deviation data, and performs data annotation; The data correction and governance module clarifies the degree of deviation of the current data and matches the corresponding correction measures for data governance.

[0020] Embodiment 2, as Figure 2As shown, based on Example 1, the present invention provides a technical solution: Preferably, in the deviation trend analysis module, the calculation process of the data deviation value is: Traverse the analysis results and historical data containing data deviation in the data fusion sequence, obtain relevant information and features of each entity in different time periods, combine historical data and business needs, select fixed evaluation cycles and data ranges from historical data, determine benchmark data including standard deviation, skewness and kurtosis features as benchmark data for data governance, where benchmark data is the average value of each feature in historical data, compare and analyze the differences between each feature and its corresponding benchmark data in each data point one by one, calculate feature deviation values, integrate each feature deviation value, and comprehensively calculate the data deviation value. Analyze the calculated data deviation value to clarify the degree of data deviation and distribution, identify the data deviation trend by analyzing the change trend of the deviation value over time, use chart tools (line charts) to display the change of the deviation value over time, observe the key nodes and turning points in the deviation trend, and analyze the causes and impacts of the deviation; Furthermore, the calculation expression of the data deviation value is: ; In the formula, is the data deviation value, which is used to measure the degree of data deviation. is the standard deviation eigenvalue, which indicates the degree of dispersion of the data. is the benchmark data of the standard deviation, that is, the average value of the standard deviation in the historical data. is the skewness eigenvalue, which indicates the asymmetry of the data distribution. is the benchmark data of skewness, that is, the average value of skewness in historical data, is the kurtosis eigenvalue, which indicates the sharpness of the data distribution. is the benchmark data of kurtosis, that is, the average value of kurtosis in historical data, , , They represent the minimum values ​​of the standard deviation, skewness, and kurtosis eigenvalues, respectively, and are used for normalization to ensure that the terms in the formula are comparable. The value range is between 0 and 1. When it is close to 0, it means that the deviation between the data and the benchmark data is very small and the data is relatively stable. When it is close to 1, it means that the data deviates greatly from the benchmark data and there is a significant deviation in the data. As the difference between each feature in the data point and its corresponding benchmark data increases, the data deviation value will gradually increase and tend to 1. On the contrary, when the difference between the data point and the benchmark data is small, will tend to 0; In the data annotation analysis module, the data annotation process is as follows: Obtain the calculated data deviation value from the deviation trend analysis module, use the data deviation value to analyze the difference between each data point and the benchmark data, measure the data deviation, and preset different deviation thresholds to determine the deviation data in combination with business needs and data deviation values. The deviation data is divided into different deviation levels, namely, slight deviation, medium deviation and severe deviation. The calculated data deviation value is compared with the preset deviation threshold, and the deviation level of the data point is marked according to the comparison result, and the marked data is stored in different data sets for subsequent processing; Furthermore, multiple deviation levels are set with multiple deviation thresholds in one-to-one correspondence, specifically satisfying the following relationship: Slight deviations: ; When the data deviation value When the data falls within this range, it means that the difference between the data and the benchmark data is small and the degree of deviation is mild. For data with slight deviation, simple correction measures can be taken, such as fine-tuning the parameters of the data acquisition equipment, or appropriate weighting can be performed on the data during data analysis to reduce the impact of the deviation on the analysis results. Medium deviation: ; When the data deviation value When the data falls within this range, it means that the difference between the data and the benchmark data is relatively obvious, and the degree of deviation is medium. For data with medium deviation, more detailed analysis and correction are required. Combined with business knowledge and data characteristics, the cause of the deviation should be found and corresponding correction measures should be taken, including correcting the data processing logic, re-collecting some data, etc. Serious deviations: ; When the data deviation value When the data falls within this range, it means that the data is significantly different from the benchmark data and the degree of deviation is serious. For data with serious deviations, it is necessary to attach great importance to it and deal with it urgently. Immediately stop using these data for analysis and decision-making, organize a professional team to conduct in-depth investigation and analysis, find out the root cause of the deviation, and take strong corrective measures, including re-collecting a large amount of data and comprehensively revising the data processing process. in, is the data deviation value, is the upper threshold corresponding to slight deviation and the lower threshold corresponding to medium deviation, The upper threshold corresponding to moderate deviation and the lower threshold corresponding to severe deviation; In the data correction management module, the data management process is as follows: Obtain annotated deviation data from the data annotation and analysis module, including the characteristics of the data points, the degree of deviation, and the deviation grade. According to the annotation information of the deviation data, clarify the degree of deviation of each data point. Based on the different phenomena of deviation data in historical data, build a correction measure library containing predefined solutions for different degrees and types of deviations to implement data governance for deviation data. According to the assessed degree of deviation, match the corresponding correction measures from the correction measure library, and apply the selected correction measures to the deviation data. The correction measures include data resampling, data interpolation, data smoothing, data conversion, and correction of data processing logic. For deviations caused by data collection errors, re-collect data to obtain more accurate data information. For deviations caused by missing values ​​or outliers, use interpolation methods to fill or correct them. For deviations caused by large data fluctuations, smoothing algorithms are used to smooth the data to reduce the impact of short-term fluctuations. For deviations caused by uneven data distribution or skew, data conversion is performed to make the data distribution more uniform and symmetrical. For deviations caused by improper data processing, the logic and algorithms of data processing are corrected to ensure the correctness and consistency of data processing. After correction, data changes are monitored and the data deviation values ​​before and after correction are compared to evaluate the effectiveness of data governance. If the governance effect is not ideal, the cause of the deviation is re-analyzed based on the verification results, the correction measures are adjusted, and the correction operation is performed again until a satisfactory governance effect is achieved. The governed data is stored in the data warehouse to replace the original deviation data, and detailed information on data governance activities is recorded, including correction measures, data changes before and after correction, and correction time.

[0021] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent data management system based on a large model, characterized by: The intelligent data management system includes a data acquisition module, a data cleaning and fusion module, a model training module, a deviation trend analysis module, a data annotation analysis module and a data correction management module, wherein the modules are connected by electrical signals; The data acquisition module is used to acquire basic data from a data source for implementing data governance, wherein the basic data includes historical data and current data; The data cleaning and fusion module pre-processes the collected basic data and integrates the data to obtain a data fusion sequence; The model training module trains a deviation analysis model based on historical data in the data fusion sequence to analyze the deviation of the data; The deviation trend analysis module determines the baseline data for data governance and calculates the data deviation value to clarify the degree of deviation by combining the analysis results of the data deviation situation with historical data; The data annotation analysis module presets a deviation threshold based on the deviation degree analysis result, determines the deviation degree of the deviation data, and performs data annotation; The data correction and management module clarifies the degree of deviation of the current data and matches corresponding correction measures for data management.

2. The intelligent data management system based on a large model according to claim 1 is characterized by: In the data acquisition module, the basic data acquisition process is as follows: According to the goals and needs of data governance, identify the types of data sources that need to be connected to implement data governance, including internal business systems, external partner systems, social media platforms, and IoT devices, and establish connections with the corresponding data sources; Define data extraction rules according to business needs and data governance strategies, determine the data fields, data formats and data frequencies to be extracted, and set the priority and sequence of data extraction. According to the defined rules, extract basic data including historical data and current data from the connected data sources; For historical data, batch collection is used to obtain the required data stored in the database at one time. For current data, real-time collection is used to continuously obtain data through real-time data streams. Convert the collected data into a unified format and standardize the data to eliminate differences between different data sources.

3. The intelligent data management system based on a large model according to claim 2 is characterized by: In the data cleaning and fusion module, the integration and acquisition process of the data fusion sequence is as follows: Import the collected basic data into the data cleaning and fusion module, and conduct preliminary analysis on the data to clarify the structure, type, distribution and potential problems of the data; Scan the data set integrated with basic data and perform data cleaning, including missing value processing, outlier detection and duplicate data processing. According to the characteristics of the data and business needs, identify the fields and records where the missing values ​​are located, and implement processing measures to fill in or delete the missing values; Horizontally integrate data records of the same entity from different data sources, and integrate data records of different time periods in the same data source to form a continuous time series data set; Create a unique ID number for each set of data, and establish an association between each data and the ID number based on the timestamp association attribute between the data, and integrate them to form a data fusion sequence.

4. The intelligent data management system based on a large model according to claim 3 is characterized by: In the model training module, the analysis process of data deviation is as follows: Extracting the historical data part from the data fusion sequence as the input of training the deviation analysis model, and selecting the support vector machine model to train the deviation analysis model; Perform feature analysis on historical data, extract features for deviation analysis, including standard deviation, skewness, and kurtosis, and integrate the features to obtain a feature data set; The feature data set is divided into a training set and a test set. The deviation analysis model is trained using the training set data combined with the support vector machine model. The trained model is evaluated using the test set to calculate the model's accuracy, recall, and F1 score evaluation indicators. Use the trained model to perform deviation analysis on the data in the data fusion sequence, identify the deviation in the data, predict whether the data has deviation, explain the identified deviation, and analyze the cause and impact of the deviation.

5. The intelligent data management system based on a large model according to claim 4 is characterized in that: In the deviation trend analysis module, the calculation process of the data deviation value is: Traverse the analysis results and historical data containing data deviation in the data fusion sequence to obtain relevant information and features of each entity in different time periods; Combine historical data with business needs, select a fixed evaluation period and data range from historical data, and determine benchmark data including standard deviation, skewness, and kurtosis characteristics as the benchmark data for data governance. The benchmark data is the average value of each characteristic in the historical data. Compare and analyze the differences between each feature in each data point and its corresponding benchmark data one by one, and calculate the feature deviation value; Integrate the deviation values ​​of each feature, calculate the data deviation value comprehensively, analyze the calculated data deviation value, and clarify the deviation degree and distribution of the data; By analyzing the changing trend of deviation values ​​over time, the deviation trend of the data can be identified, and chart tools can be used to display the changes of deviation values ​​over time. The key nodes and turning points in the deviation trend can be observed, and the causes and impacts of the deviation can be analyzed.

6. The intelligent data management system based on a large model according to claim 5 is characterized by: The calculation expression of the data deviation value is: ； In the formula, is the data deviation value, is the standard deviation eigenvalue, is the benchmark data of standard deviation, is the skewness eigenvalue, is the benchmark data of skewness, is the kurtosis eigenvalue, is the benchmark data of kurtosis, , , Respectively represent the minimum values ​​of the standard deviation, skewness and kurtosis eigenvalues, The value range is between 0 and 1.

7. The intelligent data management system based on a large model according to claim 6 is characterized by: In the data annotation analysis module, the data annotation process is as follows: Obtain the calculated data deviation value from the deviation trend analysis module, use the data deviation value to analyze the difference between each data point and the benchmark data, and measure the data deviation; Based on business needs and data deviation values, different deviation thresholds are preset to determine deviation data, and deviation data is divided into different deviation levels, namely slight deviation, medium deviation and severe deviation; The calculated data deviation value is compared with the preset deviation threshold, and the deviation level of the data point is marked according to the comparison result, and the marked data is then stored in different data sets.

8. The intelligent data management system based on a large model according to claim 7 is characterized by: A plurality of the deviation levels are provided with a plurality of the deviation thresholds in one-to-one correspondence, specifically satisfying the following relationship: Slight deviations: ; Medium deviation: ; Serious deviations: ; in, is the data deviation value, is the upper threshold corresponding to slight deviation and the lower threshold corresponding to medium deviation, The upper threshold for moderate deviations and the lower threshold for severe deviations.

9. The intelligent data management system based on a large model according to claim 8 is characterized by: In the data correction management module, the process of data management is as follows: Obtain annotated deviation data from the data annotation analysis module, including the characteristics of the data point, the degree of deviation, and the degree of deviation, and clarify the degree of deviation of each data point based on the annotation information of the deviation data; Based on different phenomena of deviation data in historical data, a correction measure library containing predefined solutions for different deviation degrees and types is built to implement data governance for deviation data; According to the assessed degree of deviation, matching corresponding correction measures from a correction measure library, and applying the selected correction measures to the deviation data, wherein the correction measures include data resampling, data interpolation, data smoothing, data conversion, and modifying data processing logic; After correction, monitor the data changes and compare the data deviation values ​​before and after correction to evaluate the effect of data governance. If the governance effect is not ideal, re-analyze the cause of the deviation based on the verification results, adjust the correction measures, and perform the correction operation again until a satisfactory governance effect is achieved; Store the governed data in the data warehouse, replace the original deviation data, and record the detailed information of the data governance activities, including corrective measures, data changes before and after correction, and correction time.

Images