Digital management system and method based on multi-source data information

Through adaptive data interface, self-supervised learning algorithms and intelligent data conversion tools, automatic annotation and distributed storage technology, the problem of inefficient data processing in smart city projects is solved, and efficient data management and automated processing are achieved.

CN118626771BActive Publication Date: 2024-11-22BITA (SHANGHAI) DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410759183.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-22
Estimated Expiration
2044-06-13

AI Technical Summary

Technical Problem

When facing complex and diverse data in smart city projects, existing digital management methods require a lot of manual intervention and configuration work, which is inefficient, lacks flexibility in data interfaces, and is difficult to automatically adapt to new data sources, and is processing unstructured data. Time annotation and conversion rely on manual labor, inefficient and error-prone.

Method used

Adaptive data interface and self-supervised learning algorithm are used to automatically identify and clean noise and outliers in the data, and the data standardization processing is realized through intelligent data conversion tools; the self-supervised learning model is used to automatically label unstructured data to generate data conversion Rules, reduce manual intervention; adopt distributed storage technology and full-text retrieval technology to improve data storage and retrieval efficiency; build a federated learning platform for data collaborative analysis to ensure data privacy and security; deploy smart contracts for data quality monitoring and traceability.

Benefits of technology

It realizes the automated configuration of data interfaces and the rapid adaptation of new data sources, significantly improves the automation and efficiency of data processing, improves data quality and processing efficiency, reduces manual intervention, improves the automation of labeling and conversion, and meets the satisfaction of The need for large-scale data management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118626771B_ABST
    Figure CN118626771B_ABST
Patent Text Reader

Abstract

The present invention discloses a digital management system and method based on multi-source data information, and relates to the technical field of digital information management. The present invention realizes automatic configuration of data interfaces and rapid adaptation of new data sources through adaptive data interfaces and automatic pattern extraction, reduces manual intervention, and improves the automation and efficiency of data processing; uses a self-supervised learning model to automatically identify and clean noise and abnormal values ​​in data, and realizes automatic standardization of data through a pre-training model, which significantly improves data quality and processing efficiency; at the same time, the self-supervised learning model is used to automatically annotate unstructured data, and data conversion rules are generated according to the annotation results, data in different formats are unified, and the automation of annotation and conversion is improved; and distributed storage technology is adopted, and an efficient data indexing and retrieval mechanism is established based on full-text retrieval technology, which significantly improves data storage and retrieval efficiency and meets the needs of large-scale data management.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of information digital management, and in particular to a multi-source data-based information digital management system and method. Background Art

[0002] Digital management refers to the use of computer, communication, network and other technologies to quantify management objects and management behaviors through statistical techniques to achieve management activities and methods of R&D, planning, organization, production, coordination, sales, service, innovation and other functions.

[0003] For example, the patent with publication number CN115935705B discloses a digital management system and method for industrial control information based on multi-source data, which relates to the field of industrial control management technology; the system includes a multi-source data acquisition module, a hierarchical division module, a prediction model construction and analysis module, a data extraction and analysis module, an optimization and adjustment module and a feedback management module; the output end of the multi-source data acquisition module is connected to the input end of the hierarchical division module; the output end of the hierarchical division module is connected to the input end of the prediction model construction and analysis module; the output end of the prediction model construction and analysis module is connected to the input end of the data extraction and analysis module; the output end of the data extraction and analysis module is connected to the input end of the optimization and adjustment module; the output end of the optimization and adjustment module is connected to the input end of the feedback management module; the invention also provides a digital management method for industrial control information based on multi-source data for specific analysis.

[0004] Smart city projects need to integrate a large number of heterogeneous data sources from different departments and systems, such as traffic data, environmental data, energy data, etc., to achieve unified data management and analysis. However, traditional digital management methods cannot meet the expected requirements when faced with complex and diverse data in smart city projects. Whenever a new data source is added, a lot of manual intervention and configuration work is required, which is inefficient. The data interface lacks flexibility and is difficult to automatically adapt to new data sources, resulting in high system maintenance costs. At the same time, when processing unstructured data, labeling and conversion often rely on manual labor, which is inefficient and error-prone. Therefore, there is an urgent need for a digital management system and method based on multi-source data information to solve such problems. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a digital management system and method based on multi-source data information, which solves the problems that the digital management methods in the prior art require a lot of manual intervention and configuration work when facing complex and diverse data in smart city projects, are inefficient, and the data interface lacks flexibility. At the same time, when processing unstructured data, labeling and conversion often rely on manual labor, which is inefficient and prone to errors.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] The present invention provides a digital management method based on multi-source data information, comprising:

[0008] Step 1. Collect urban project data, identify data sources, and identify multi-source data from various departments and systems in smart cities, including traffic sensor data, environmental monitoring data, energy consumption data, and social media data. The data interface uses an adaptive data interface to automatically adapt to newly added data sources and reduce manual configuration. A self-supervised learning algorithm is used to automatically identify and clean noise and outliers in the data, and an intelligent data conversion tool is used to automatically standardize the data according to the pre-trained model.

[0009] Step 2. Data annotation conversion: Use a self-supervised learning model to automatically annotate unstructured data, including text and images. Use the self-supervised learning model to pre-train a large amount of unlabeled data to generate a feature extraction and annotation model. Based on the automatic annotation results, generate data conversion rules to unify data in different formats, thereby optimizing data conversion rules and reducing manual intervention.

[0010] Step 3. Data storage management: using distributed storage technology to store data, and establishing data indexing and retrieval mechanisms based on full-text retrieval technology to improve data retrieval efficiency;

[0011] Step 4. Collaborative data analysis, building a federated learning platform, each institution trains the model locally, only shares the model parameters, and does not transmit the original data. Differential privacy and secure multi-party computing technologies are used to ensure data privacy and security. The central server aggregates the model parameters uploaded by each institution and updates the global model. Through multiple rounds of iterations, the performance and accuracy of the global model are gradually improved. Different institutions in the smart city project need to collaboratively analyze their own data to obtain more comprehensive insights. In order to protect data privacy, a federated learning platform is used. Each institution trains the model locally, only shares the model parameters, and does not transmit the original data. The central server is responsible for aggregating the model parameters uploaded by each institution and updating the global model.

[0012] Step 5. Data quality monitoring and traceability, deploy smart contracts, and perform automated data quality checks and audits; generate hash values ​​at each stage of data generation, transmission, and processing, and record them in the blockchain to ensure data integrity and traceability; users can query the data generation and processing history through the blockchain traceability function.

[0013] The present invention is further configured as follows: in step 1, the adaptive data interface is constructed in the following manner:

[0014] Define each data source D i The eigenvector X i, including structured data, semi-structured data and unstructured data. Structured data includes the number of fields and data type distribution; semi-structured data includes the number of nodes and nesting levels; unstructured data includes text length and number of keywords. i =[x i1 ,x i2 ,...x ij ], where x ij is the jth feature of the i-th data source, and j also represents the number of features;

[0015] Use support vector machine SVM to train the classification model. The decision function of the classifier C of the classification model is: Where X represents the feature vector of the data to be classified, α j is the weight of the support vector, y j is the label of the support vector, K(X,X j ) represents the kernel function, which is used to calculate the similarity between feature vectors, and b is the bias term;

[0016] The training process includes: collecting labeled training data, extracting feature vectors, and training the SVM model;

[0017] Perform pattern extraction to identify the structure and format information of the data source. Suppose the data source D i The mode is S i ;

[0018] For structured data, extract table structure and field information: i ={(Field j ,tod j )j=1,2,...,n}, where Field j ,tod j Represents the field name and data type of the jth feature respectively; for semi-structured data, extracts node and attribute information: S i ={Node j |j=1,2,...,n}, where Node j The node name representing the jth feature;

[0019] For unstructured data, natural language processing (NLP) technology is used to extract text features: i ={Keyw j |j=1,2,...,n}, where Keyw j The keyword representing the jth feature;

[0020] According to the extracted pattern S i , generate data transformation rules T i ;

[0021] The present invention is further configured as follows: in step 1, constructing the adaptive data interface method further includes:

[0022] The data conversion rules are expressed using context-free grammar CFG: G = (N, S, P, S), where N is a set of non-terminal symbols, S is a set of terminal symbols, P is a set of generation rules, and S represents a start symbol;

[0023] Apply the generated transformation rule T i , the data source data D i Convert to target format data D i '; thereby building an adaptive data interface that automatically adapts to newly added data sources and improves the automation and efficiency of data processing;

[0024] The present invention is further configured as follows: the data standardization processing method in step 1 is:

[0025] Initially clean the data, remove null values ​​and duplicate data, extract features from the original data, and assume that the time series data set is X = {x1, x2, ...x i}, where x i Represents the feature vector of the i-th sample;

[0026] Self-supervised learning models are trained by predicting partial information of the data to identify noise and outliers;

[0027] For each sample, construct the model input and use the BERT model to convert the time series data into the model format: Input(x i )=[CLS]+Tokenize(x i )+[SEP], where Tokenize(x i ) indicates the conversion of feature vector into model input, [CLS] and [SEP] indicate the start and end of input respectively;

[0028] The mask language model MLM is used for mask operation. Suppose the input sequence is Randomly select some marks to mask:

[0029] The training is performed by maximizing the predicted probability of the masked mark. is the model for the masked label x ij The predicted probability, the loss function is defined as: Where L MLM is the loss function of the masked language model, n is the number of samples,

[0030] masked is the index set of masked marks. represents the probability predicted by the model;

[0031] The total loss function is combined with the masked language model loss, that is, L = L MLM , where L represents the total loss function;

[0032] Use the optimization algorithm Adam to minimize the total loss function and update the model parameters: θ * =argminθL,θ * Represents the model parameters after training;

[0033] Detect noise and outliers in data using trained self-supervised learning models;

[0034] For each sample, calculate the residual between the model's prediction and the true value: residual i represents the residual of the i-th sample;

[0035] Calculate the residuals and consider the residuals exceeding the threshold as outliers;

[0036] Then use intelligent data conversion tools to automatically standardize data based on pre-trained models;

[0037] Perform mean normalization and Zscore standardization, and then use the pre-trained model to standardize the new data;

[0038] This enables the self-supervised learning algorithm to identify and clean noise and outliers, as well as data standardization;

[0039] The present invention is further configured as follows: the data standardization processing method in step 1 is:

[0040] Initially clean the data, remove null values ​​and duplicate data, extract features from the original data, and assume that the time series data set is X = {x1, x2, ...x i}, where x i Represents the feature vector of the i-th sample;

[0041] Self-supervised learning models are trained by predicting partial information of the data to identify noise and outliers;

[0042] For each sample, construct the model input and use the BERT model to convert the time series data into the model format: Input(x i )=[CLS]+Tokenize(x i )+[SEP], where Tokenize(x i ) indicates the conversion of feature vector into model input, [CLS] and [SEP] indicate the start and end of input respectively;

[0043] The mask language model MLM is used for mask operation. Suppose the input sequence is Randomly select some marks to mask:

[0044] The training is performed by maximizing the predicted probability of the masked mark. is the model for the masked label x ij The predicted probability, the loss function is defined as: Where L MLM is the loss function of the masked language model, n is the number of samples,

[0045] masked is the index set of masked marks. represents the probability predicted by the model;

[0046] The total loss function is combined with the masked language model loss, that is, L = L MLM , where L represents the total loss function;

[0047] Use the optimization algorithm Adam to minimize the total loss function and update the model parameters: θ * =argminθL,θ * Represents the model parameters after training;

[0048] Detect noise and outliers in data using trained self-supervised learning models;

[0049] For each sample, calculate the residual between the model's prediction and the true value: residual i represents the residual of the i-th sample;

[0050] Calculate the residuals and consider the residuals exceeding the threshold as outliers;

[0051] Then use intelligent data conversion tools to automatically standardize data based on pre-trained models;

[0052] Perform mean normalization and Zscore standardization, and then use the pre-trained model to standardize the new data;

[0053] This enables the self-supervised learning algorithm to identify and clean noise and outliers, as well as data standardization;

[0054] The present invention is further configured as follows: the data annotation conversion method in step 2 is:

[0055] Automatically annotate unstructured data using trained self-supervised learning models;

[0056] Segment the text data, add special tags, and use the trained model to predict the masked tags to obtain the annotation results;

[0057] Generate data conversion rules based on automatic annotation results to unify data in different formats;

[0058] Extract data patterns from annotation results, analyze automatic annotation results, and extract data patterns and structural information;

[0059] Generate data transformation rules based on the extracted patterns and define transformation rules based on the data patterns;

[0060] Then, according to the generated conversion rules, the data in different formats are unified and the generated conversion rules are applied to convert the data;

[0061] Convert the data format according to the conversion rules;

[0062] The present invention is further configured as follows: the data indexing and retrieval mechanism in step 3 is specifically:

[0063] Split the large file F into k blocks, each block is of size B, that is, F = {B1, B2, ..., B k};

[0064] Each block is replicated r times on multiple nodes: R(B i )={B i1 ,B i2 ,...,B ir}, where R(B i ) indicates data block B i A collection of copies of

[0065] From each document i Extract the word w j , for each word w j Create an inverted index, I(w j )={d i |w j ∈Terms(d i )}, where Terms(d i ) indicates document d i The word collection in Terms(d i )={w i1 ,w i2 ,...w im};

[0066] Calculate TFIDF weights: Where f(w,d) represents the number of occurrences of word w in document d;

[0067] Where N represents the total number of documents, |{d∈D|w∈d} represents the number of documents containing word w;

[0068] tf-idf(w,d,D)=tf(w,d)×idf(w,D);

[0069] Document vector representation:

[0070] The present invention is further configured as follows: the data indexing and retrieval mechanism in step 3 further includes:

[0071] Perform similarity calculation and retrieval, cosine similarity: where q·d i represents the dot product of the query vector and the document vector, ||q||||d i || represents the vector norm;

[0072] Perform distributed indexing and retrieval, split the document collection D into n subsets, each subset D i Indexing is performed on different nodes, D = {D1, D2, ... D n};

[0073] Each node processes the query q independently, and then aggregates and sorts the results to get R i (q);

[0074] The final result R(q) is the union of the results returned by all nodes:

[0075] The present invention is further configured as follows: the method for constructing a federated learning platform for collaborative data analysis in step 4 is:

[0076] Build a central server, i.e. the coordination server, and multiple clients, i.e. the participating institutions;

[0077] Each participating institution has local computing resources for model training;

[0078] The central server initializes the global model and sends its parameters to all participating institutions;

[0079] Each participating institution preprocesses and standardizes local data and uses the received global model parameters for training locally;

[0080] The training data is the internal data of the organization, and the model parameters are updated during the training process;

[0081] After training is completed, each participating institution sends the locally updated model parameters to the central server;

[0082] The central server receives local model parameters from each participating institution;

[0083] The central server performs weighted aggregation on the received model parameters, and broadcasts the updated global model parameters to all participating institutions as the initial parameters for the next round of training;

[0084] Repeat the local training and parameter aggregation process until the model converges;

[0085] The present invention also provides a digital management system based on multi-source data information, including:

[0086] The data collection module includes: a data source identification unit, which is used to identify multi-source data from various departments and systems in the smart city, including traffic sensor data, environmental monitoring data, energy consumption data, and social media data; ensuring comprehensive coverage of data sources and providing a basis for subsequent analysis;

[0087] Adaptive data interface, automatically adapting to newly added data sources;

[0088] The data annotation conversion module includes: an automatic annotation unit, which uses a self-supervised learning model to automatically annotate unstructured data; segment text data and add special tags, use the trained model to predict the masked tags, and obtain the annotation results; generate data conversion rules based on the automatic annotation results to unify data in different formats; extract data patterns from the annotation results, analyze the automatic annotation results, and extract data patterns and structural information; generate data conversion rules based on the extracted patterns; unify data in different formats based on the generated conversion rules, and apply the generated conversion rules to convert the data;

[0089] The data storage module includes: a distributed storage unit, which uses distributed storage technology to store data and improve the reliability and scalability of data storage;

[0090] A data index retrieval unit, providing a data index retrieval function;

[0091] Data collaborative analysis module, including: federated learning platform for data collaborative analysis;

[0092] The data monitoring module includes: a smart contract unit that deploys smart contracts, generates hash values ​​at each stage of data generation, transmission and processing, and records them in the blockchain, records the state changes and hash values ​​of data at each processing stage, and provides a data traceability interface.

[0093] Compared with the prior art, the present invention has the following beneficial effects:

[0094] The present invention realizes automatic configuration of data interfaces and rapid adaptation of new data sources through adaptive data interfaces and automatic pattern extraction, reduces manual intervention, and improves the automation and efficiency of data processing; uses a self-supervised learning model to automatically identify and clean noise and outliers in data, and realizes automatic standardization of data through a pre-trained model, significantly improving data quality and processing efficiency; at the same time, the self-supervised learning model is used to automatically annotate unstructured data, and data conversion rules are generated according to the annotation results to unify data in different formats, reduce manual intervention, and improve the automation of annotation and conversion; and adopts distributed storage technology, and establishes an efficient data indexing and retrieval mechanism based on full-text retrieval technology, significantly improving data storage and retrieval efficiency, and meeting the needs of large-scale data management;

[0095] The digital management methods in the existing technology solve the problems that when facing complex and diverse data in smart city projects, a lot of manual intervention and configuration work is required, which is inefficient and the data interface lacks flexibility. At the same time, when processing unstructured data, labeling and conversion often rely on manual work, which is inefficient and error-prone. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 This is a flow chart of the digital management method based on multi-source data information of the present invention. DETAILED DESCRIPTION

[0097] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0098] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0099] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0100] Example 1

[0101] See also Figure 1 The present invention provides a digital management method based on multi-source data information, comprising:

[0102] Step 1. Collect urban project data, identify data sources, and identify multi-source data from various departments and systems in smart cities, including traffic sensor data, environmental monitoring data, energy consumption data, and social media data; the data interface uses an adaptive data interface to automatically adapt to newly added data sources and reduce manual configuration; a self-supervised learning algorithm is used to automatically identify and clean noise and outliers in the data, and an intelligent data conversion tool is used to automatically standardize data according to the pre-trained model. The adaptive data interface is constructed in the following way:

[0103] Define each data source D i The eigenvector X i , including structured data, semi-structured data and unstructured data. Structured data includes the number of fields and data type distribution; semi-structured data includes the number of nodes and nesting levels; unstructured data includes text length and number of keywords. i =[x i1 ,x i2 ,...x ij ], where x ij is the jth feature of the i-th data source, and j also represents the number of features;

[0104] Use support vector machine SVM to train the classification model. The decision function of the classifier C of the classification model is: Where X represents the feature vector of the data to be classified, α j is the weight of the support vector, y j is the label of the support vector, K(X,X j ) represents the kernel function, which is used to calculate the similarity between feature vectors, and b is the bias term;

[0105] The training process includes: collecting labeled training data, extracting feature vectors, and training the SVM model;

[0106] Perform pattern extraction to identify the structure and format information of the data source. Suppose the data source D i The mode is S i ;

[0107] For structured data, extract table structure and field information: i ={(Field j ,tod j )|j=1,2,...,n}, where Fieldj ,tod j Respectively represent the field name and data type of the j-th feature;

[0108] For semi-structured data, extract node and attribute information: S i ={Node j |j=1,2,...,n}, where Node j The node name representing the jth feature;

[0109] For unstructured data, natural language processing (NLP) technology is used to extract text features: i ={Keyw j |j=1,2,...,n}, where Keyw j The keyword representing the jth feature;

[0110] According to the extracted pattern S i , generate data transformation rules T i ; Use context-free grammar CFG to represent data conversion rules: G = (N, S, P, S), where N is a set of non-terminal symbols, S is a set of terminal symbols, P is a set of generation rules, and S represents a start symbol;

[0111] Apply the generated transformation rule T i , the data source data D i Convert to target format data D i '; thereby building an adaptive data interface that automatically adapts to newly added data sources and improves the automation and efficiency of data processing;

[0112] The data standardization process is as follows:

[0113] Initially clean the data, remove null values ​​and duplicate data, extract features from the original data, and assume that the time series data set is X = {x1, x2, ...x i}, where x i Represents the feature vector of the i-th sample;

[0114] Self-supervised learning models are trained by predicting partial information of the data to identify noise and outliers;

[0115] For each sample, construct the model input and use the BERT model to convert the time series data into the model format: Input(x i )=[CLS]+Tokenize(x i )+[SEP], where Tokenize(x i ) indicates the conversion of feature vector into model input, [CLS] and [SEP] indicate the start and end of input respectively;

[0116] The mask language model MLM is used for mask operation. Suppose the input sequence is Randomly select some marks to mask:

[0117] The training is performed by maximizing the predicted probability of the masked mark. is the model for the masked label x ij The predicted probability, the loss function is defined as: Where L MLM is the loss function of the masked language model, n is the number of samples,

[0118] masked is the index set of masked marks. represents the probability predicted by the model;

[0119] The total loss function is combined with the masked language model loss, that is, L = L MLM , where L represents the total loss function;

[0120] Use the optimization algorithm Adam to minimize the total loss function and update the model parameters: θ * =argminθL,θ * Represents the model parameters after training;

[0121] Detect noise and outliers in data using trained self-supervised learning models;

[0122] For each sample, calculate the residual between the model's prediction and the true value: residual i represents the residual of the i-th sample;

[0123] Calculate the residuals and consider the residuals exceeding the threshold as outliers;

[0124] Then use intelligent data conversion tools to automatically standardize data based on pre-trained models;

[0125] Perform mean normalization and Zscore standardization, and then use the pre-trained model to standardize the new data;

[0126] This enables the self-supervised learning algorithm to identify and clean noise and outliers, as well as data standardization;

[0127] Step 2. Data annotation conversion: Use a self-supervised learning model to automatically annotate unstructured data, including text and images. Use the self-supervised learning model to pre-train a large amount of unlabeled data to generate a feature extraction and annotation model. Based on the automatic annotation results, generate data conversion rules to unify data in different formats, thereby optimizing data conversion rules and reducing manual intervention. The self-supervised learning model for automatic annotation is constructed as follows:

[0128] Collect unlabeled text data, perform preliminary cleaning, segment and tokenize each sample, and construct input sequences;

[0129] The masked language model MLM is used to randomly mask some tags, and the next sentence prediction NSP task is deployed to construct continuous sentence pairs.

[0130] The model architecture uses the pre-trained model BERT, which includes multi-layer bidirectional Transformers, for model training;

[0131] Define the loss function combining MLM and NSP, and use the optimization algorithm to minimize the total loss function;

[0132] The data annotation conversion method is:

[0133] Automatically annotate unstructured data using trained self-supervised learning models;

[0134] Segment the text data, add special tags, and use the trained model to predict the masked tags to obtain the annotation results;

[0135] Generate data conversion rules based on automatic annotation results to unify data in different formats;

[0136] Extract data patterns from annotation results, analyze automatic annotation results, and extract data patterns and structural information;

[0137] Generate data transformation rules based on the extracted patterns and define transformation rules based on the data patterns;

[0138] Then, according to the generated conversion rules, the data in different formats are unified and the generated conversion rules are applied to convert the data;

[0139] Convert the data format according to the conversion rules;

[0140] Step 3. Data storage management: Distributed storage technology is used for data storage, and a data index and retrieval mechanism is established based on full-text retrieval technology to improve data retrieval efficiency; the data index and retrieval mechanism is specifically as follows:

[0141] Split the large file F into k blocks, each block is of size B, that is, F = {B1, B2, ..., Bk};

[0142] Each block is replicated r times on multiple nodes: R(B i )={B i1 ,B i2 ,...,B ir}, where R(B i ) indicates data block B i A collection of copies of

[0143] From each document i Extract the word w j , for each word w j Create an inverted index, I(w j )={d i |w j ∈Terms(d i )}, where Terms(d i ) indicates document d i The word collection in Terms(d i )={w i1 ,w i2 ,...w im};

[0144] Calculate TFIDF weights: Where f(w,d) represents the number of occurrences of word w in document d;

[0145] Where N represents the total number of documents, |{d∈D|w∈d} represents the number of documents containing word w;

[0146] tf-idf(w,d,D)=tf(w,d)×idf(w,D);

[0147] Document vector representation:

[0148] Perform similarity calculation and retrieval, cosine similarity: where q·d i represents the dot product of the query vector and the document vector, ||q||||d i || represents the vector norm;

[0149] Perform distributed indexing and retrieval, split the document collection D into n subsets, each subset D i Indexing is performed on different nodes, D = {D1, D2, ... D n};

[0150] Each node processes the query q independently, and then aggregates and sorts the results to get R i (q);

[0151] The final result R(q) is the union of the results returned by all nodes: Realize efficient data storage management and fast retrieval mechanism;

[0152] Step 4. Collaborative data analysis, building a federated learning platform, each institution trains the model locally, only shares the model parameters, and does not transmit the original data. Differential privacy and secure multi-party computing technologies are used to ensure data privacy and security. The central server aggregates the model parameters uploaded by each institution and updates the global model. Through multiple rounds of iterations, the performance and accuracy of the global model are gradually improved. Different institutions in the smart city project need to collaboratively analyze their own data to obtain more comprehensive insights. In order to protect data privacy, a federated learning platform is used. Each institution trains the model locally, only shares the model parameters, and does not transmit the original data. The central server is responsible for aggregating the model parameters uploaded by each institution and updating the global model.

[0153] The method of building a federated learning platform for collaborative data analysis in step 4 is as follows:

[0154] Build a central server, i.e. the coordination server, and multiple clients, i.e. the participating institutions;

[0155] Each participating institution has local computing resources for model training;

[0156] The central server initializes the global model and sends its parameters to all participating institutions;

[0157] Each participating institution preprocesses and standardizes local data and uses the received global model parameters for training locally;

[0158] The training data is the internal data of the organization, and the model parameters are updated during the training process;

[0159] After training is completed, each participating institution sends the locally updated model parameters to the central server;

[0160] The central server receives local model parameters from each participating institution;

[0161] The central server performs weighted aggregation on the received model parameters, and broadcasts the updated global model parameters to all participating institutions as the initial parameters for the next round of training;

[0162] Repeat the local training and parameter aggregation process until the model converges;

[0163] Step 5. Data quality monitoring and traceability, deploy smart contracts, and perform automated data quality checks and audits; generate hash values ​​at each stage of data generation, transmission, and processing, and record them in the blockchain to ensure data integrity and traceability; users can query the data generation and processing history through the blockchain traceability function;

[0164] The present invention also provides a digital management system based on multi-source data information, including:

[0165] Data acquisition module, including:

[0166] The data source identification unit is used to identify multi-source data from various departments and systems in the smart city, including traffic sensor data, environmental monitoring data, energy consumption data, and social media data; ensuring comprehensive coverage of data sources and providing a basis for subsequent analysis;

[0167] Adaptive data interface, automatically adapting to newly added data sources;

[0168] Data annotation conversion module, including:

[0169] The automatic annotation unit uses a self-supervised learning model to automatically annotate unstructured data; segment text data and add special tags, use the trained model to predict masked tags, and obtain annotation results; generate data conversion rules based on the automatic annotation results to unify data in different formats; extract data patterns from the annotation results, analyze the automatic annotation results, and extract data patterns and structural information; generate data conversion rules based on the extracted patterns; unify data in different formats based on the generated conversion rules, and apply the generated conversion rules to convert the data;

[0170] Data storage module, including:

[0171] Distributed storage unit, which uses distributed storage technology for data storage to improve the reliability and scalability of data storage;

[0172] A data index retrieval unit, providing a data index retrieval function;

[0173] Data collaborative analysis module, including:

[0174] Federated learning platform for collaborative data analysis;

[0175] Data monitoring module, including:

[0176] The smart contract unit deploys smart contracts, generates hash values ​​at each stage of data generation, transmission and processing, and records them in the blockchain, records the state changes and hash values ​​of the data at each processing stage, and provides a data traceability interface.

[0177] The present invention realizes automatic configuration of data interface and rapid adaptation of new data source through adaptive data interface and automatic pattern extraction, reduces manual intervention, and improves the automation degree and efficiency of data processing; utilizes self-supervised learning model to automatically identify and clean noise and outliers in data, and realizes automatic standardized processing of data through pre-training model, which significantly improves data quality and processing efficiency; at the same time, unstructured data is automatically labeled through self-supervised learning model, and data conversion rules are generated according to the labeling results, which unifies data of different formats, reduces manual intervention, and improves the automation degree of labeling and conversion; and adopts distributed storage technology, and establishes efficient data indexing and retrieval mechanism based on full-text retrieval technology, which significantly improves data storage and retrieval efficiency and meets the needs of large-scale data management.

[0178] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A digital management method based on multi-source data information, characterized in that: include: Step 1. Collect urban project data, identify data sources, and identify multi-source data from various departments and systems in smart cities, including traffic sensor data, environmental monitoring data, energy consumption data, and social media data. The data interface uses an adaptive data interface to automatically adapt to newly added data sources. A self-supervised learning algorithm is used to automatically identify and clean noise and outliers in the data, and an intelligent data conversion tool is used to automatically perform data standardization processing based on the pre-trained model. Step 2. Data annotation conversion: Use a self-supervised learning model to automatically annotate unstructured data, including text and images; generate data conversion rules based on the automatic annotation results to unify data in different formats; Step 3. Data storage management: using distributed storage technology to store data and establishing data indexing and retrieval mechanisms based on full-text retrieval technology; Step 4. Collaborative data analysis, building a federated learning platform, each institution trains the model locally, only shares the model parameters, and does not transmit the original data; the central server aggregates the model parameters uploaded by each institution and updates the global model; Step 5. Data quality monitoring and tracing, deployment of smart contracts, automated data quality checks and audits; hash values ​​are generated at each stage of data generation, transmission and processing, and recorded in the blockchain; In step 1, the adaptive data interface is constructed as follows: Define each data source D i The eigenvector X i , including structured data, semi-structured data and unstructured data, X i =[x i1 ,x i2 ,...x ij ], where x ij is the jth feature of the i-th data source, and j also represents the number of features; Use support vector machine SVM to train the classification model. The decision function of the classifier C of the classification model is: Where X represents the feature vector of the data to be classified, α j is the weight of the support vector, y j is the label of the support vector, K(X,X j ) represents the kernel function, which is used to calculate the similarity between feature vectors, and b is the bias term; The training process includes: collecting labeled training data, extracting feature vectors, and training the SVM model; Perform pattern extraction to identify the structure and format information of the data source. Suppose the data source D i The mode is S i ; For structured data, extract table structure and field information: i ={(Field j ,tod j )|j=1,2,...,n}, where Field j ,tod j Represents the field name and data type of the jth feature respectively; for semi-structured data, extracts node and attribute information: S i ={Node j |j=1,2,...,n}, where Node j The node name representing the jth feature; For unstructured data, natural language processing (NLP) technology is used to extract text features: i ={Keyw j |j=1,2,...,n}, where Keyw j The keyword representing the jth feature; According to the extracted pattern S i , generate data transformation rules T i ; In step 1, the method of constructing an adaptive data interface also includes: The data conversion rules are represented by context-free grammar CFG: G = (N, Σ, P, S), where N is a set of non-terminal symbols, Σ is a set of terminal symbols, P is a set of generation rules, and S represents a start symbol; Apply the generated transformation rule T i , the data source data D i Convert to target format data D i ′; The data standardization processing method in step 1 is: Initially clean the data, remove null values ​​and duplicate data, extract features from the original data, and assume that the time series data set is X = {x1, x2, ...x i }, where x i Represents the feature vector of the i-th sample; For each sample, construct the model input and use the BERT model to convert the time series data into the model format: Input(x i )=[CLS]+Tokenize(x i )+[SEP], where Tokenize(x i ) indicates the conversion of feature vector into model input, [CLS] and [SEP] indicate the start and end of input respectively; The mask language model MLM is used for mask operation. Suppose the input sequence is Randomly select some marks to mask: The training is performed by maximizing the predicted probability of the masked mark. is the model for the masked label x ij The predicted probability, the loss function is defined as: Where L MLM is the loss function of the masked language model, n is the number of samples, masked is the index set of masked marks. represents the probability predicted by the model; The total loss function is combined with the masked language model loss, that is, L = L MLM , where L represents the total loss function; Use the optimization algorithm Adam to minimize the total loss function and update the model parameters: θ * = arg min θ L, θ * Represents the model parameters after training; Detect noise and outliers in data using trained self-supervised learning models; For each sample, calculate the residual between the model's prediction and the true value: residual i represents the residual of the i-th sample; Calculate the residuals and consider the residuals exceeding the threshold as outliers; Then use intelligent data conversion tools to automatically standardize data based on pre-trained models; Perform mean normalization and Zscore standardization, and then use the pre-trained model to standardize the new data.

2. The method for digital management of multi-source data information according to claim 1, characterized in that: The self-supervised learning model for automatic labeling in step 2 is constructed as follows: Collect unlabeled text data, perform preliminary cleaning, segment and tokenize each sample, and construct input sequences; The masked language model MLM is used to randomly mask some tags, and the next sentence prediction NSP task is deployed to construct continuous sentence pairs. The model architecture uses the pre-trained model BERT, which includes multi-layer bidirectional Transformers, for model training; Define the loss function that combines MLM and NSP, and use the optimization algorithm to minimize the total loss function.

3. The method for digital management of multi-source data information according to claim 2, characterized in that: The data annotation conversion method in step 2 is: Automatically annotate unstructured data using trained self-supervised learning models; Segment the text data, add special tags, and use the trained model to predict the masked tags to obtain the annotation results; Generate data conversion rules based on automatic annotation results to unify data in different formats; Extract data patterns from annotation results, analyze automatic annotation results, and extract data patterns and structural information; Generate data transformation rules based on the extracted patterns and define transformation rules based on the data patterns; Then, according to the generated conversion rules, the data in different formats are unified and the generated conversion rules are applied to convert the data; Convert the data format according to the conversion rules.

4. The method for digital management of multi-source data information according to claim 3, characterized in that: The data indexing and retrieval mechanism in step 3 is as follows: Split the large file F into k blocks, each block is of size B, that is, F = {B1, B2, ..., B k }; Each block is replicated r times on multiple nodes: R(B i )={B i1 ,B i2 ,...,B ir }, where R(B i ) indicates data block B i A collection of copies of From each document i Extract the word w j , for each word w j Create an inverted index. Among them, Terms(d i ) indicates document d i The word collection in Terms(d i )={w i1 ,w i2 ,...w im }; Calculate TFIDF weights: Where f(w,d) represents the number of occurrences of word w in document d; Where N represents the total number of documents, |{d∈D|w∈d}| represents the number of documents containing word w; tf-idf(w,d,D)=tf(w,d)×idf(w,D); Document vector representation:

5. The method for digital management of multi-source data information according to claim 4, characterized in that: The data indexing and retrieval mechanism in step 3 also includes: Perform similarity calculation and retrieval, cosine similarity: where q·d i represents the dot product of the query vector and the document vector, ||q||||d i || represents the vector norm; Perform distributed indexing and retrieval, split the document collection D into n subsets, each subset D i Indexing is performed on different nodes, D = {D1, D2, ... D n }; Each node processes the query q independently, and then aggregates and sorts the results to get R i (q); The final result R(q) is the union of the results returned by all nodes:

6. The method for digital management of multi-source data information according to claim 5, characterized in that: The method of building a federated learning platform for collaborative data analysis in step 4 is as follows: Build a central server, i.e. the coordination server, and multiple clients, i.e. the participating institutions; Each participating institution has local computing resources for model training; The central server initializes the global model and sends its parameters to all participating institutions; Each participating institution preprocesses and standardizes local data and uses the received global model parameters for training locally; The training data is the internal data of the organization, and the model parameters are updated during the training process; After training is completed, each participating institution sends the locally updated model parameters to the central server; The central server receives local model parameters from each participating institution; The central server performs weighted aggregation on the received model parameters, and broadcasts the updated global model parameters to all participating institutions as the initial parameters for the next round of training; The local training and parameter aggregation process is repeated until the model converges.

7. Based on the digital management system of multi-source data information, it is characterized by: include: The data collection module includes: a data source identification unit for identifying multi-source data from various departments and systems in the smart city, including traffic sensor data, environmental monitoring data, energy consumption data, and social media data; The adaptive data interface automatically adapts to the newly added data source, uses the self-supervised learning algorithm to automatically identify and clean the noise and outliers in the data, and uses the intelligent data conversion tool to automatically standardize the data according to the pre-trained model. The adaptive data interface is constructed as follows: define each data source D i The eigenvector X i , including structured data, semi-structured data and unstructured data, X i =[x i1 ,x i2 ,...x ij ], where x ij is the jth feature of the i-th data source, and j also represents the number of features; Use support vector machine SVM to train the classification model. The decision function of the classifier C of the classification model is: Where X represents the feature vector of the data to be classified, α j is the weight of the support vector, y j is the label of the support vector, K(X,X j ) represents the kernel function, which is used to calculate the similarity between feature vectors, and b is the bias term; The training process includes: collecting labeled training data, extracting feature vectors, and training the SVM model; Perform pattern extraction to identify the structure and format information of the data source. Suppose the data source D i The mode is S i ; For structured data, extract table structure and field information: i ={(Field j ,tod j )|j=1,2,...,n}, where Field j ,tod j Represents the field name and data type of the jth feature respectively; for semi-structured data, extracts node and attribute information: S i ={Node j |j=1,2,...,n}, where Node j The node name representing the jth feature; For unstructured data, natural language processing (NLP) technology is used to extract text features: i ={Keyw j |j=1,2,...,n}, where Keyw j The keyword representing the jth feature; According to the extracted pattern S i , generate data transformation rules T i , The data conversion rules are represented by context-free grammar CFG: G = (N, Σ, P, S), where N is a set of non-terminal symbols, Σ is a set of terminal symbols, P is a set of generation rules, and S represents a start symbol; Apply the generated transformation rule T i , the data source data D i Convert to target format data D i ′, The data standardization process is as follows: Initially clean the data, remove null values ​​and duplicate data, extract features from the original data, and assume that the time series data set is X = {x1, x2, ...x i }, where x i Represents the feature vector of the i-th sample; For each sample, construct the model input and use the BERT model to convert the time series data into the model format: Input(x i )=[CLS]+Tokenize(x i )+[SEP], where Tokenize(x i ) indicates the conversion of feature vector into model input, [CLS] and [SEP] indicate the start and end of input respectively; The mask language model MLM is used for mask operation. Suppose the input sequence is Randomly select some marks to mask: x′ i =[x i1 ,MASK,x i3 ,...,MASK]; The training is performed by maximizing the predicted probability of the masked mark. is the model for the masked label x ij The predicted probability, the loss function is defined as: Where L MLM is the loss function of the masked language model, n is the number of samples, masked is the index set of masked marks. represents the probability predicted by the model; The total loss function is combined with the masked language model loss, that is, L = L MLM , where L represents the total loss function; Use the optimization algorithm Adam to minimize the total loss function and update the model parameters: θ * = arg min θ L, θ * Represents the model parameters after training; Detect noise and outliers in data using trained self-supervised learning models; For each sample, calculate the residual between the model's prediction and the true value: residual i represents the residual of the i-th sample; Calculate the residuals and consider the residuals exceeding the threshold as outliers; Then use intelligent data conversion tools to automatically standardize data based on pre-trained models; Perform mean normalization and Zscore standardization, and then use the pre-trained model to standardize the new data; The data annotation conversion module includes: an automatic annotation unit, which uses a self-supervised learning model to automatically annotate unstructured data; segment text data and add special tags, use the trained model to predict the masked tags, and obtain the annotation results; generate data conversion rules based on the automatic annotation results to unify data in different formats; extract data patterns from the annotation results, analyze the automatic annotation results, and extract data patterns and structural information; generate data conversion rules based on the extracted patterns; unify data in different formats based on the generated conversion rules, and apply the generated conversion rules to convert the data; The data storage module includes: a distributed storage unit, which uses distributed storage technology to store data; A data index retrieval unit, providing a data index retrieval function; Data collaborative analysis module, including: federated learning platform for data collaborative analysis; The data monitoring module includes: a smart contract unit that deploys smart contracts, generates hash values ​​at each stage of data generation, transmission and processing, and records them in the blockchain, records the state changes and hash values ​​of data at each processing stage, and provides a data traceability interface.