DMS data management system

Abstract

The invention discloses a DMS data management system, and relates to the technical field of data management systems, and the system comprises a sparsity evaluation module which is used for carrying out the sparsity evaluation of a comprehensive data set in the DMS data management system, and outputting a data sparsity evaluation result; the uncertainty evaluation module is used for performing dynamic Bayesian estimation and uncertainty quantification on the comprehensive data set in the DMS data management system and outputting an uncertainty quantification evaluation result; the acquisition strategy optimization module is used for optimizing an acquisition strategy according to the data sparse evaluation result and the uncertainty quantitative evaluation result; and the data hierarchical acquisition module is used for performing data dependency detection on the acquired data, scoring the data source through a preset data quality evaluation algorithm, and performing data hierarchical acquisition according to a descending order of scores of the data source. By introducing an accuracy and uncertainty evaluation module, a data acquisition strategy is optimized, data quality control is enhanced, and the influence of data uncertainty on an analysis result is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data management systems, in particular to a DMS data management system. Background Art

[0002] With the rapid development of big data technology, data collection, management and analysis are becoming more and more important. However, existing data management systems often have the following problems when processing complex data sets:

[0003] Insufficient data accuracy assessment: Existing data management systems lack effective evaluation models and are unable to accurately measure the overall accuracy and uncertainty of data sets, which can easily lead to misjudgments during data processing and affect the reliability of data decisions.

[0004] Inefficient data collection: Traditional data collection methods do not effectively optimize and hierarchically process data sources and data features, resulting in low efficiency in data collection and analysis. In particular, when processing large or dynamic data sets, it is often difficult to meet real-time requirements.

[0005] Limited means of data set optimization: Existing systems often lack effective optimization strategies when dealing with data set complexity and are unable to perform targeted optimization collection based on the characteristics and uncertainties of different data sources, resulting in high data set redundancy and low data quality. Summary of the invention

[0006] Based on the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a DMS data management system to solve the above-mentioned technical problems.

[0007] To achieve the above object, the present invention provides the following technical solution: DMS data management system, comprising:

[0008] Sparsity evaluation module: used to evaluate the sparsity of the comprehensive data set in the DMS data management system and output the data sparsity evaluation results;

[0009] Uncertainty assessment module: used to perform dynamic Bayesian estimation and uncertainty quantification on the comprehensive data set in the DMS data management system, and output uncertainty quantification assessment results;

[0010] An acquisition strategy optimization module is used to optimize the acquisition strategy according to the data sparsity evaluation result and the uncertainty quantification evaluation result;

[0011] Data layered collection module: used to detect the data dependency of the collected data, score the data source through a preset data quality assessment algorithm, and perform data layered collection in descending order of the data source score.

[0012] The present invention is further configured that the sparsity evaluation module includes a sampling sparsity evaluation unit, a sampling divergence evaluation unit and a data sparsity evaluation result unit;

[0013] Sampling sparsity evaluation unit: used to apply low sampling rate detection algorithm to comprehensive data sets to evaluate the sampling sparsity of data dimensions;

[0014] Sampling divergence evaluation unit: for evaluating the sampling divergence of the data dimension whose sampling sparsity value is lower than the preset sparsity threshold according to the preset sampling divergence analysis;

[0015] Data sparsity evaluation result unit: used to output the sampling sparsity and the sampling divergence as a data sparsity evaluation result.

[0016] The present invention is further configured such that the calculation logic of the low sampling rate detection algorithm is: Among them, Ss i is the sampling sparsity of the i-th data dimension, T is the length of the time series, γ is the adjustment factor used to control the sensitivity of the nonlinear function, x ij is the data value of the i-th data dimension at the j-th time series sample point, μ ij is the center value of the i-th data dimension at the j-th time series sample point;

[0017] The calculation logic of the preset sampling divergence analysis is: Among them, Di i is the sampling divergence of the i-th data dimension, p ij is the actual distribution probability, representing the observation frequency of the i-th data dimension at the j-th time series sample point, q ij is the theoretical expected probability, representing the ideal data distribution, and ∈1 is a small amount to avoid the singularity of the logarithmic function.

[0018] The present invention is further configured such that the uncertainty assessment module includes a prior distribution construction module, a Bayesian update module and an uncertainty quantification module;

[0019] A prior distribution construction module is used to construct a prior distribution for the comprehensive data set in the DMS data management system according to the data dimension;

[0020] A Bayesian update module, used to apply a dynamic Bayesian update algorithm according to the prior distribution, adjust the parameter estimation of the data dimension in real time, and construct a posterior distribution;

[0021] The uncertainty quantification module is used to calculate the uncertainty of the data dimension according to the posterior distribution.

[0022] The present invention is further configured such that the construction logic of the prior distribution is: Among them, P(θ i ) is the prior distribution of the i-th data dimension, Z i is a normalization constant used to ensure that the sum of the prior distribution is 1, Ω i is the time series sample space of the i-th data dimension, f(x ij ,θ i ) is the data value of the i-th data dimension at the j-th time series sample point and the parameter θ of the i-th data dimension i The energy function of

[0023] The construction logic of the posterior distribution is: Among them, P(θ i |X i ) is the posterior distribution of the i-th data dimension, P(X i |θ i ) is the likelihood function of the i-th data dimension, indicating that given the parameter θ i When X is observed i The probability of i is the parameter space of the i-th data dimension;

[0024] The calculation logic of the uncertainty of the data dimension is: Among them, Uc(θ i ) is the uncertainty of the i-th data dimension.

[0025] The present invention is further configured to optimize the acquisition strategy, including:

[0026] The sampling sparsity and uncertainty are weighted and summed to obtain a collection priority score, and the collection is performed in descending order of the collection priority score;

[0027] The collection volume of the data dimension is dynamically adjusted according to the collection priority score. The calculation logic of the collection volume is: Among them, A i is the collection amount of the i-th data dimension, Fp(i) is the collection priority score of the i-th data dimension, A 总 The total amount of available collection resources.

[0028] The present invention is further configured that the data layered collection module includes a dependency detection unit, a data source scoring unit and a data layered collection unit;

[0029] A dependency detection unit, used to construct a data dependency matrix based on the collected data;

[0030] A data source scoring unit, used to score the data source according to the data dependency matrix using a preset data quality assessment algorithm;

[0031] The data stratification collection unit is used to collect data in stratification in descending order of the scores of the data sources.

[0032] The present invention is further configured such that the construction logic of the data dependency matrix is: Among them, D(k,i) is the data dependency matrix, x ki is the value of the kth data source in the i-th data dimension, x mi is the value of the mth data source in the i-th data dimension, M is the number of data sources excluding the kth data source, Ω i is the adopted space of the i-th data dimension, ∈2 is a small value introduced to avoid the singularity of the logarithmic function;

[0033] The logic for scoring data sources is: Among them, Qs k is the score of the kth data source, T is the length of the time series, and λ is the adjustment factor used to control the impact of dependency on the score.

[0034] The present invention is further configured such that the logic of hierarchical collection is: Among them, A k A is the amount of data collected from the kth data source for the i-th data dimension, i is the collection amount of the i-th data dimension.

[0035] The present invention provides a DMS data management system, including: a sparsity evaluation module: used to perform sparsity evaluation on a comprehensive data set in the DMS data management system, and output a data sparsity evaluation result; an uncertainty evaluation module: used to perform dynamic Bayesian estimation and uncertainty quantification on the comprehensive data set in the DMS data management system, and output an uncertainty quantification evaluation result; an optimization acquisition strategy module: used to optimize the acquisition strategy according to the data sparsity evaluation result and the uncertainty quantification evaluation result; a data hierarchical acquisition module: used to perform data dependency detection on the collected data, score the data source through a preset data quality evaluation algorithm, and perform data hierarchical acquisition according to the descending order of the score of the data source, and the beneficial effects produced include:

[0036] 1. Improve the accuracy of data accuracy assessment: The present invention introduces an accuracy assessment module and an uncertainty assessment module, which can dynamically evaluate the comprehensive data set, avoiding the errors caused by a single assessment method in traditional methods. By quantifying data uncertainty, more accurate assessment results can be provided, thereby improving the credibility and reliability of the data;

[0037] 2. Optimize the data collection process: By optimizing the collection strategy module, the present invention can collect and process data sets from different sources in layers according to the data accuracy and uncertainty evaluation results. This method not only reduces the generation of redundant data, but also greatly improves the efficiency of data collection. It is especially suitable for processing dynamic and large-scale data sets, greatly reducing the time cost of collection and processing;

[0038] 3. Enhanced data quality control: The present invention provides a mechanism that combines data stratification collection and data quality assessment, which can automatically monitor and evaluate data quality, and adjust data collection strategies in real time according to fluctuations in data quality to ensure high-quality data input and output, thereby providing a reliable data foundation for the data analysis process.

[0039] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0041] Figure 1 The figure is a schematic diagram of the structure of a DMS data management system according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0044] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0045] DMS data management system, such as Figure 1 As shown, including:

[0046] Sparsity evaluation module: used to evaluate the sparsity of the comprehensive data set in the DMS data management system and output the data sparsity evaluation results;

[0047] Uncertainty assessment module: used to perform dynamic Bayesian estimation and uncertainty quantification on the comprehensive data set in the DMS data management system, and output uncertainty quantification assessment results;

[0048] An acquisition strategy optimization module is used to optimize the acquisition strategy according to the data sparsity evaluation result and the uncertainty quantification evaluation result;

[0049] Data layered collection module: used to detect the data dependency of the collected data, score the data source through a preset data quality assessment algorithm, and perform data layered collection in descending order of the data source score.

[0050] Specifically, the sparsity evaluation module is used to perform sparsity evaluation on the comprehensive data set in the DMS data management system. The main task is to identify the sparse areas in the data set, that is, the areas with low sampling frequency or insufficient samples. The evaluation results are used for subsequent strategy optimization; the present invention is further configured that the sparsity evaluation module includes a sampling sparsity evaluation unit, a sampling divergence evaluation unit and a data sparsity evaluation result unit;

[0051] Sampling sparsity evaluation unit: used to apply a low sampling rate detection algorithm to the comprehensive data set to evaluate the sampling sparsity of the data dimension; the present invention is further configured such that the calculation logic of the low sampling rate detection algorithm is: Among them, Ss i is the sampling sparsity of the i-th data dimension, T is the length of the time series, γ is the adjustment factor used to control the sensitivity of the nonlinear function, x ij is the data value of the i-th data dimension at the j-th time series sample point, μ ijis the center value of the data dimension at the jth time series sample point of the ith data dimension; specifically, the above calculation logic is used to evaluate the sampling sparsity of the data dimension, and quantify the sparsity of the sampling by detecting the deviation between the sample points of different time series and the center value. The higher the sparsity, the less sampled the data dimension is, and the data may not be complete. The calculation of sampling sparsity is achieved through a nonlinear function, which can accurately reflect the distribution of data at different time points. The sparsity of each data dimension is determined by the difference between the observed value in the time series and the center value of the dimension. The larger the difference, the higher the sparsity may be. Through nonlinear functions It can effectively amplify the impact of sparsely sampled areas and quantify the sparsity of sampling. Furthermore, the adjustment factor γ is used to control the sensitivity of the nonlinear function to data differences. The value range is 0.1 to 10. The larger the γ value, the more sensitive the function is to data differences.

[0052] Sampling divergence evaluation unit: for evaluating the sampling divergence of the data dimension whose sampling sparsity value is lower than the preset sparsity threshold according to the preset sampling divergence analysis; the calculation logic of the preset sampling divergence analysis is: Among them, Di i is the sampling divergence of the i-th data dimension, p ij is the actual distribution probability, representing the observation frequency of the i-th data dimension at the j-th time series sample point, q ij is the theoretical expected probability, representing the ideal data distribution, and ∈1 is a small amount to avoid the singularity of the logarithmic function. Specifically, the above calculation logic calculates the sampling divergence of each data dimension by comparing the difference between the actual sampling distribution and the expected distribution. The sampling divergence is used to evaluate whether the data of a certain dimension deviates from the expected sampling distribution. The degree of deviation of the data is quantified by accumulating the logarithmic difference between the actual distribution probability and the theoretical distribution probability at each time series point. The sampling divergence evaluation is used to quantify the difference between the actual sampling and the theoretical distribution of the data dimension. By comparing the actual sampling distribution p ij and the expected distribution q ij , evaluates the degree of deviation of the data dimension from the ideal distribution at different time points; further, ∈1 takes a very small value, including 10 -6 , in order to avoid the denominator being zero when calculating the logarithmic function, there is no restriction on the specific value here; by performing logarithmic processing and divergence calculation on the data distribution, those sample points with significant deviations can be effectively amplified, helping the system to identify abnormal sampling points or data noise, thereby further improving data quality.

[0053] Data sparsity evaluation result unit: used to output the sampling sparsity and the sampling divergence as data sparsity evaluation results. Specifically, the output here can be the weighted sum of the sampling sparsity and the sampling divergence, or can be output as a data pair. By combining the sampling sparsity and the sampling divergence, the unit can comprehensively evaluate the sparsity problem of the data, identify the lack of data frequency, and find the imbalance in data distribution, so as to comprehensively measure the data quality;

[0054] The present invention is further configured such that the uncertainty assessment module includes a prior distribution construction module, a Bayesian update module and an uncertainty quantification module;

[0055] The prior distribution construction module is used to construct a prior distribution for the comprehensive data set in the DMS data management system according to the data dimension; the present invention is further configured such that the construction logic of the prior distribution is: Among them, P(θ i ) is the prior distribution of the i-th data dimension, Z i is a normalization constant used to ensure that the sum of the prior distribution is 1, Ω i is the time series sample space of the i-th data dimension, f(x ij ,θ i ) is the data value of the i-th data dimension at the j-th time series sample point and the parameter θ of the i-th data dimension i Energy function; Specifically, the above calculation logic is used to construct a prior distribution for the comprehensive data set in the DMS data management system and determine the initial probability distribution of each data dimension. In dynamic Bayesian estimation, the prior distribution is the basis of the entire reasoning process and represents the system's initial assumptions about the data model parameters when there is no observed data. The energy function is used to define the relationship between data and model parameters, and the normalization constant is combined to ensure that the sum of the probability distribution is 1, and finally the prior distribution of each data dimension is generated; further, the parameter θ i is the model parameter, which represents the set of prior parameters of the i-th data dimension. According to the specific application scenario, θ i It includes a combination of parameters such as mean, variance, and regression coefficient. The range of values ​​depends on the distribution characteristics of the data and is not specifically restricted here. The prior distribution reflects the system's initial assumptions about the parameters of each data dimension, which helps guide the system on how to infer parameters in the absence of data and provides a basis for subsequent dynamic adjustments.

[0056] The Bayesian update module is used to apply the dynamic Bayesian update algorithm according to the prior distribution, adjust the parameter estimation of the data dimension in real time, and construct the posterior distribution; the construction logic of the posterior distribution is: Among them, P(θ i |Xi ) is the posterior distribution of the i-th data dimension, P(X i |θ i ) is the likelihood function of the i-th data dimension, indicating that given the parameter θ i When X is observed i The probability of i is the parameter space of the ith data dimension; specifically, the above calculation logic is based on the principle of Bayesian reasoning, combining the prior distribution with the observed data, dynamically adjusting the parameter estimates of each data dimension, and finally constructing the posterior distribution. This module updates the system's inference of data in real time through Bayes' theorem, especially when new data arrives, it can adjust previous assumptions and estimates based on the latest observations. The posterior distribution represents the system's correction of the uncertainty of the model parameters after combining the observed data; the likelihood function depends on the relationship between the observed data and the parameters. Its value range is 0 to 1, indicating that given the parameter θ i Next, we observe data X i The larger the value, the better the fit between the parameter and the observed data. Bayesian update can dynamically adjust the model parameters according to real-time observed data. With the arrival of new data, the system can continuously optimize parameter estimates to make the model more accurate and flexible. The Bayesian update module provides the system with a mechanism to deal with data uncertainty and noise. Even when the data is sparse or incomplete, the system can still make reasonable inferences based on prior knowledge to reduce the uncertainty of the model.

[0057] The uncertainty quantification module is used to calculate the uncertainty of the data dimension according to the posterior distribution; the calculation logic of the uncertainty of the data dimension is: Among them, Uc(θ i ) is the uncertainty of the i-th data dimension; specifically, the above calculation logic is based on the Bayesian posterior distribution to quantify the uncertainty of the data dimension. The core of this module is to use entropy as a metric to evaluate the degree of uncertainty in parameter estimation. By i |X i ) can give the uncertainty quantification result of each data dimension. The larger the entropy value, the better the model is at determining the parameter θ. i The higher the uncertainty of the estimate, the smaller the entropy value, which means the model is more certain about the estimate of the parameter.

[0058] The present invention is further configured to optimize the acquisition strategy, including:

[0059] The sampling sparsity and uncertainty are weighted and summed to obtain the collection priority score, and the collection is performed in descending order according to the collection priority score; specifically, through the weighted collection priority score, the system can dynamically adjust the order of data collection, give priority to collecting dimensions with high scores, and help the system collect the most important data with limited collection resources, thereby improving the overall data coverage and completeness.

[0060] The collection volume of the data dimension is dynamically adjusted according to the collection priority score. The calculation logic of the collection volume is: Among them, A i is the collection amount of the i-th data dimension, Fp(i) is the collection priority score of the i-th data dimension, A 总 is the total amount of available collection resources. Specifically, the above logic is used to dynamically adjust the amount of data collected for each dimension according to the collection priority score of each data dimension. By weighting the available resources according to the priority score, the system can better allocate limited collection resources and give priority to collecting data dimensions with high scores. The ultimate goal is to optimize the data collection strategy and ensure that resources are reasonably allocated to the dimensions that need them most; according to the collection priority score Fp(i) of each data dimension, the available collection resources A 总 Assigned to each data dimension A i , and reasonably allocate resources according to the weight of the score, ensuring that collection resources are allocated first to high-scoring dimensions. By dynamically allocating according to the collection priority score, the system can allocate limited collection resources to the most needed dimensions, maximizing the improvement of data quality; the system takes into account both the sparsity of data sampling and the uncertainty of the model on the data dimension, and reasonably allocates collection resources through weighted scoring to ensure comprehensive data coverage and model stability.

[0061] The present invention is further configured that the data layered collection module includes a dependency detection unit, a data source scoring unit and a data layered collection unit;

[0062] The dependency detection unit is used to construct a data dependency matrix based on the collected data; the present invention is further configured such that the construction logic of the data dependency matrix is: Among them, D(k,i) is the data dependency matrix, x ki is the value of the kth data source in the i-th data dimension, x mi is the value of the mth data source in the i-th data dimension, M is the number of data sources excluding the kth data source, Ω iis the adopted space of the i-th data dimension, ∈2 is the tiny value introduced to avoid the singularity of the logarithmic function; specifically, the above calculation logic constructs a data dependency matrix to evaluate the interdependence of different data sources in a specific data dimension. By comparing the collection values ​​of multiple data sources in the same data dimension, the matrix quantifies the degree of dependence between data sources. The higher the degree of dependence, the stronger the dependence of a data source on other data sources in this dimension, which reflects the independence or redundancy of the data; through the dependency matrix, the system can identify which data sources depend on other data sources and which data sources are more independent. This is of great significance for optimizing data collection and reducing redundant data; data sources with higher dependence can reduce the frequency of collection, thereby reducing the resource consumption of the system. On the contrary, independent data sources with lower dependence can be retained or the collection frequency can be increased first to ensure the diversity and independence of data.

[0063] The data source scoring unit is used to score the data source according to the data dependency matrix using a preset data quality assessment algorithm; the logic for scoring the data source is: Among them, Qs k is the score of the kth data source, T is the length of the time series, and λ is the adjustment factor used to control the impact of dependency on the score; specifically, the above calculation logic scores the quality of each data source based on the constructed data dependency matrix, and the score quantifies the independence, integrity and reliability of the data source based on data dependency. The purpose of the score is to give higher scores to data sources with strong independence and high data quality, and lower scores to data sources with strong dependence by considering the degree of dependence of the data source on different time series. The scoring results are used to guide subsequent collection strategies and data processing optimization; the adjustment factor λ is used to adjust the impact of dependency on the score. A larger λ will make the score more sensitive to changes in dependency, and the value range is 0.1 to 10. By scoring the dependency of the data source, the system can clearly identify the independence and reliability of the data source. Data sources with strong independence and low dependence can get higher scores, which helps guide data collection strategies.

[0064] The data layered collection unit is used to collect data in layers according to the descending order of the scores of the data sources; the present invention is further configured such that the logic of layered collection is: Among them, A k A is the amount of data collected from the kth data source for the i-th data dimension, i is the amount of data collected in the i-th data dimension. Specifically, the above calculation logic is based on the score Qs of each data source. k , and collect data from each data source in descending order. The system divides the total amount of collected resources A into iAllocate resources to different data sources in proportion. Data sources with higher scores will be allocated more resources and will be collected first. The purpose of this logic is to ensure that the system prioritizes the collection of data sources with strong independence and high data quality, thereby improving the reliability and effectiveness of the overall data; by allocating resources based on the score, the system can prioritize the collection of high-quality, independent data sources, ensure that resources are used most effectively, and improve the quality of overall data collection; automatically reduce data collection from low-scoring data sources, reduce the collection of redundant data, optimize the efficiency of data storage and processing, and realize dynamic hierarchical collection of data sources. The above mechanism is flexible and dynamic, and can adjust the collection strategy in real time according to the score, providing an efficient resource allocation solution for complex data management systems.

[0065] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0066] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0067] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0068] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0069] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0071] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0074] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0075] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. DMS data management system, characterized by: include: Sparsity evaluation module: used to evaluate the sparsity of the comprehensive data set in the DMS data management system and output the data sparsity evaluation results; Uncertainty assessment module: used to perform dynamic Bayesian estimation and uncertainty quantification on the comprehensive data set in the DMS data management system, and output uncertainty quantification assessment results; An acquisition strategy optimization module is used to optimize the acquisition strategy according to the data sparsity evaluation result and the uncertainty quantification evaluation result; Data layered collection module: used to detect the data dependency of the collected data, score the data source through a preset data quality assessment algorithm, and perform data layered collection in descending order of the data source score.

2. The DMS data management system according to claim 1, characterized in that: The sparsity evaluation module includes a sampling sparsity evaluation unit, a sampling divergence evaluation unit and a data sparsity evaluation result unit; Sampling sparsity evaluation unit: used to apply low sampling rate detection algorithm to comprehensive data sets to evaluate the sampling sparsity of data dimensions; Sampling divergence evaluation unit: used for evaluating the sampling divergence of the data dimension according to the preset sampling divergence analysis for the data of the data dimension whose sampling sparsity value is lower than the preset sparsity threshold; Data sparsity evaluation result unit: used to output the sampling sparsity and the sampling divergence as a data sparsity evaluation result.

3. The DMS data management system according to claim 2, characterized in that: The calculation logic of the low sampling rate detection algorithm is: Among them, Ss i is the sampling sparsity of the i-th data dimension, T is the length of the time series, γ is the adjustment factor used to control the sensitivity of the nonlinear function, x ij is the data value of the i-th data dimension at the j-th time series sample point, μ ij is the center value of the i-th data dimension at the j-th time series sample point; The calculation logic of the preset sampling divergence analysis is: Among them, Di i is the sampling divergence of the i-th data dimension, p ij is the actual distribution probability, representing the observation frequency of the i-th data dimension at the j-th time series sample point, q ij is the theoretical expected probability, representing the ideal data distribution, and ∈1 is a small amount to avoid the singularity of the logarithmic function.

4. The DMS data management system according to claim 2, characterized in that: The uncertainty assessment module includes a prior distribution construction module, a Bayesian update module and an uncertainty quantification module; A prior distribution construction module is used to construct a prior distribution for the comprehensive data set in the DMS data management system according to the data dimension; A Bayesian update module, used to apply a dynamic Bayesian update algorithm according to the prior distribution, adjust the parameter estimation of the data dimension in real time, and construct a posterior distribution; The uncertainty quantification module is used to calculate the uncertainty of the data dimension according to the posterior distribution.

5. The DMS data management system according to claim 4, characterized in that: The construction logic of the prior distribution is: Among them, P(θ i ) is the prior distribution of the i-th data dimension, Z i is a normalization constant used to ensure that the sum of the prior distribution is 1, Ω i is the time series sample space of the i-th data dimension, f(x ij ,θ i ) is the data value of the i-th data dimension at the j-th time series sample point and the parameter θ of the i-th data dimension i The energy function of The construction logic of the posterior distribution is: Among them, P(θ i |X i ) is the posterior distribution of the i-th data dimension, P(X i |θ i ) is the likelihood function of the i-th data dimension, indicating that given the parameter θ i When X is observed i The probability of i is the parameter space of the i-th data dimension; The calculation logic of the uncertainty of the data dimension is: Among them, Uc(θ i ) is the uncertainty of the i-th data dimension.

6. The DMS data management system according to claim 4, characterized in that: Optimize collection strategies, including: The sampling sparsity and uncertainty are weighted and summed to obtain a collection priority score, and the collection is performed in descending order of the collection priority score; The collection volume of the data dimension is dynamically adjusted according to the collection priority score. The calculation logic of the collection volume is: Among them, A i is the collection amount of the i-th data dimension, Fp(i) is the collection priority score of the i-th data dimension, A 总 The total amount of available collection resources.

7. The DMS data management system according to claim 1, characterized in that: The data layered collection module includes a dependency detection unit, a data source scoring unit and a data layered collection unit; A dependency detection unit, used to construct a data dependency matrix based on the collected data; A data source scoring unit, used to score the data source according to the data dependency matrix using a preset data quality assessment algorithm; The data stratification collection unit is used to collect data in stratification in descending order of the scores of the data sources.

8. The DMS data management system according to claim 7, characterized in that: The construction logic of the data dependency matrix is: Among them, D(k,i) is the data dependency matrix, x ki is the value of the kth data source in the i-th data dimension, x mi is the value of the mth data source in the i-th data dimension, M is the number of data sources excluding the kth data source, Ω i is the adopted space of the i-th data dimension, ∈2 is a small value introduced to avoid the singularity of the logarithmic function; The logic for scoring data sources is: Among them, Qs k is the score of the kth data source, T is the length of the time series, and λ is the adjustment factor used to control the impact of dependency on the score.

9. The DMS data management system according to claim 8, characterized in that: The logic of layered collection is: Among them, A k A is the amount of data collected from the kth data source for the i-th data dimension, i is the collection amount of the i-th data dimension.

Images