Internet-based data management and control system

Abstract

The invention discloses a data management and control system based on the Internet. The method comprises the following steps: step 1, establishing a database, backing up and storing data, collecting enterprise data, marking a data carrier, and transmitting the enterprise data to the management and control system; 2, carrying out standardization processing on the collected data, classifying the processed data, and extracting a training data set to carry out model training; 3, rating the data by using the trained model, and selecting a transmission node to transmit the data according to a rating result; and 4, analyzing a data relationship to construct a knowledge graph, positioning the position of the data in the data tree database by using a positioning module, identifying a data carrier mark, and identifying a data source. The method has the advantages that the data utilization efficiency is improved, and data sources can be traced.

Description

Technical Field

[0001] The present invention relates to the technical field of data management and control, and in particular to an Internet-based data management and control system. Background Art

[0002] With the rapid development of Internet technology, Internet data has exploded. A large amount of data within enterprises needs to be processed and managed in order to utilize the data. However, the efficiency of data utilization under existing technologies needs to be improved. Although existing technologies can train models and rate data through training models to determine the transmission order, when training models, the collected test data does not all meet the rating standards of the training model. There are some data that cannot be rated through the training model, and the data finally stored in the database cannot be traced to ensure the accuracy of the data source. Therefore, it is necessary to design an Internet-based data management and control system that improves data utilization efficiency and can trace data sources. Summary of the invention

[0003] The purpose of the present invention is to provide an Internet-based data management and control system to solve the problems raised in the above-mentioned background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a data management and control system based on the Internet, comprising the following steps: Step 1: Establish a database, back up and store data, collect enterprise data, mark data carriers, and transfer enterprise data to the management and control system; Step 2: Standardize the collected data, classify the processed data, and extract the training data set for model training; Step 3: Use the trained model to rate the data, and the transmission node selects a transmission node to transmit the data based on the rating results; Step 4: Analyze data relationships to build a knowledge graph, use the positioning module to locate the data in the data tree database, identify the data carrier mark, and identify the data source.

[0005] According to the above technical solution, the steps of establishing a database, backing up and storing data, collecting enterprise data, marking data carriers, and transmitting enterprise data to the management and control system include: The system uses a data interface to connect to the enterprise business system and periodically pulls data from the enterprise business system through a carrier. When the carrier pulls data, the data marking module recognizes that the system is connected to the database at this time and marks the database information on the carrier. The database module establishes a database for the data obtained for each cycle, and transmits the data to the database through the data transmission module. The data is backed up and stored in the database. The data transmission module converts the electrical signal for data transmission into a digital signal, and uses the digital signal for data transmission.

[0006] According to the above technical solution, the step of standardizing the collected data and extracting the training data set for model training includes: The standardization processing module performs standardization processing on the data, calls the first data standardization model to eliminate incomplete data, erroneous data, and duplicate data, and establishes a review database for the eliminated data to store it, connects to the review database through the data interface to obtain data, manually reviews the eliminated data, supplements the incomplete data, corrects the erroneous data, and standardizes the manually reviewed data again; Further, the second standardized model is called to process the data after cluster classification, identify the data structure of each cluster, anchor the unstructured data and semi-structured data, identify the data type in the cluster, and determine whether the data can be converted into structured data. When the data type cannot be converted into structured data, the second standardized model retains the data type. When the data type can be converted into structured data, the second standardized model converts the data into structured data. The model training module extracts historical data from the database as a training data set to train the data rating model, trains the data rating model, and adds a special rating recognition function after the data rating model training is completed to recognize the expedited transmission mark added to the data carrier.

[0007] According to the above technical solution, the step of classifying the processed data includes: Acquire the data after the first standardization processing, perform clustering processing on the data, cluster the data into structured data, semi-structured data and unstructured data, analyze the data in each cluster based on the clustering, identify the data type, classify the data according to the identified data type, and mark the data type in each classified data set.

[0008] According to the above technical solution, the step of using the trained model to rate the data, and the transmission node selecting a transmission node to transmit the data according to the rating result, includes: The data rating module obtains the data to be transmitted, calls the rating model to assess the transmission priority of the data to be transmitted, and the rating model identifies the specific data priority transmission mark on the data carrier. The rating model assesses the data as the first transmission echelon level. If the data priority transmission mark is not identified, the rating training model is used to assess the data transmission level according to the trained rules. Furthermore, the rating model transmits a message that the data to be transmitted is in the first transmission echelon level to the node management module. The node management module receives the message and starts the dedicated transmission node for expedited data to preheat the node. The expedited transmission data is transmitted through its dedicated transmission node. Furthermore, when the transmission node is receiving data for data transmission, the node management module monitors in real time the amount of remaining data to be transmitted on each transmission node. When the ratio of the remaining amount of data to be transmitted to the transmission time of the transmission node is less than a preset threshold of the system, the transmission node enters a transmission fatigue state, and the transmission speed decreases. The node management module starts a new transmission node to transmit the remaining data through the new transmission node, and the transmission node that enters the fatigue state will be shut down.

[0009] According to the above technical solution, the steps of analyzing data relationships to construct a knowledge graph, using a positioning module to locate the position of data in a data tree database, identifying a data carrier tag, and identifying a data source include: Identify the types of data clusters, identify the relationships between data in each cluster, and build a knowledge graph through the knowledge graph construction module; Furthermore, at each procedure that the data goes through, the data marking module marks the processing method of the data on the data carrier, and compares the original data after each procedure. If the comparison results are completely consistent, the next procedure will be entered. If there are differences in the comparison results, when the similarity is greater than a threshold, the system obtains the original data and continues the procedure. When the similarity is less than a threshold, the system screens out the data and hands it over to manual processing; Furthermore, the data before the difference in the comparison result is located, the data before the difference is obtained and re-entered into the program for processing, and the process the data has gone through and the source of the data are traced back according to the mark on the data carrier.

[0010] According to the above technical solution, the system includes: Data collection module, used to periodically pull data from the enterprise business system; The data processing module is used to process the data and prioritize the data through the data rating model; The data control module is used to control and manage data transmission nodes and trace abnormal data.

[0011] According to the above technical solution, the data acquisition module includes: A data collection module is used to periodically pull data through a data interface; A data transmission module, used to convert electrical signals into digital signals for data transmission; The database module is used to establish a database for the data pulled in each cycle.

[0012] According to the above technical solution, the data processing module includes: Standardization processing module, used to eliminate incomplete, erroneous, and redundant data from the collected data, and convert the data into data with a unified format; Data classification module, used to analyze data types and cluster data; Model training module, used to retrieve historical data from the historical database to train the data model; The data rating module is used to assess the transmission level of data.

[0013] According to the above technical solution, the data control module includes: A node processing module is used to manage transmission nodes; Knowledge graph construction module, used to build knowledge graphs between data; Data labeling module, used to label the data source; Data location module, used to locate the data in the knowledge graph; The data tracing module is used to trace the source of data and the process that the data has gone through.

[0014] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention, by providing a standardization processing module and a data model training module, can eliminate incomplete data, erroneous data, and duplicate data by standardizing the data, and then manually reviewing the data to ensure the integrity of the data, and by converting unstructured data and semi-structured data into structured data, it can ensure the uniformity of the data, and on this basis can improve the speed of model training, improve the accuracy of the trained model judgment, and thus improve the efficiency of data utilization, by grading the data, it can transmit the data quickly and systematically, improve the transmission speed, by controlling the transmission nodes to remain in the optimal transmission group state, further improve the data transmission efficiency, speed up resource utilization, and by analyzing the data relationship to construct a knowledge graph, it is more conducive to discovering the utilization value of the data and the hidden value of the data, and improve the utilization rate of the data, and by marking the data in each program, it is possible to trace the data processing links and the source of the data, and achieve data traceability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flowchart of a data management method based on the Internet provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the module composition of an Internet-based data management and control system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Embodiment 1: Figure 1 This is a flow chart of an Internet-based data management system provided in Embodiment 1 of the present invention. This embodiment can be applied to data management scenarios. The method can be executed by an Internet-based data management system provided in this embodiment. Figure 1 As shown, the method specifically comprises the following steps: Step 1: Establish a database, back up and store data, collect enterprise data, mark data carriers, and transfer enterprise data to the management and control system; In an embodiment of the present invention, the system utilizes a data interface to connect to an enterprise business system, and periodically pulls data from the enterprise business system through a carrier. When the carrier pulls data, a data marking module identifies that the system is connected to a database at this time, and marks the database information on the carrier. The database module establishes a database for the data acquired for each cycle, and transmits the data to the database through a data transmission module. The data is backed up and stored in the database. The data transmission module converts the electrical signal for data transmission into a digital signal, and uses the digital signal for data transmission. By establishing a database for the data acquired for each cycle, it can be ensured that the database can use the type of data collected, and will not cause data loss or data incompleteness, thereby protecting the integrity of the data. Marking the data carrier can ensure the source of the data, can determine the source of the data in the event of a system failure, trace back to complete historical data, and ensure the authenticity and integrity of the data. Converting electrical signals into digital signals can better resist interference and reduce interference to the data during transmission.

[0018] Step 2: Standardize the collected data, classify the processed data, and extract the training data set for model training; In the embodiment of the present invention, the standardization processing module performs standardization processing on the data, calls the first data standardization model to remove incomplete data, erroneous data, and duplicate data, and establishes a review database for the removed data to store it, connects to the review database through a data interface to obtain data, manually reviews the removed data, supplements the incomplete data, corrects the erroneous data, and standardizes the manually reviewed data again; Further, the data after the first standardization processing is obtained, and the data is clustered to cluster the data into structured data, semi-structured data, and unstructured data. Based on the clustering, the data in each cluster is analyzed to identify the data type, and the data is classified according to the identified data type, and the data type is marked in each classified data set; Further, the second standardized model is called to process the data after cluster classification, identify the data structure of each cluster, anchor the unstructured data and semi-structured data, identify the data type in the cluster, and determine whether the data can be converted into structured data. When the data type cannot be converted into structured data, the second standardized model retains the data type. When the data type can be converted into structured data, the second standardized model converts the data into structured data. Furthermore, the model training module extracts historical data from the database as a training data set to train the data rating model, trains the data rating model, and adds a special assessment recognition function after the data rating model training is completed to identify the expedited transmission mark added to the data carrier; by standardizing the data, incomplete data, erroneous data, and duplicate data can be eliminated, and by manually reviewing the data, the integrity of the data can be ensured. By converting unstructured data and semi-structured data into structured data, the uniformity of the data can be guaranteed. On this basis, the speed of model training can be improved, the accuracy of the trained model judgment can be improved, and the efficiency of data utilization can be improved.

[0019] Step 3: Use the trained model to rate the data, and the transmission node selects a transmission node to transmit the data based on the rating results; In an embodiment of the present invention, a data rating module obtains data to be transmitted, calls a rating model to evaluate the transmission priority of the data to be transmitted, the rating model identifies a specific data priority transmission identifier on a data carrier, and the rating model evaluates the data as a first transmission echelon level. If the data priority transmission identifier is not identified, the rating training model is used to evaluate the data transmission level according to the training rules; Furthermore, the rating model transmits a message that the data to be transmitted is in the first transmission echelon level to the node management module. The node management module receives the message and starts the dedicated transmission node for expedited data to preheat the node. The expedited transmission data is transmitted through its dedicated transmission node. Furthermore, when the transmission node is receiving data for data transmission, the node management module monitors the amount of remaining data to be transmitted on each transmission node in real time. When the ratio of the remaining amount of data to be transmitted to the transmission time of the transmission node is less than a preset threshold of the system, the transmission node enters a transmission fatigue state, and the transmission speed decreases. The node management module starts a new transmission node to transmit the remaining data through the new transmission node, and the transmission node that enters the fatigue state will be shut down. By rating the data, the data can be transmitted quickly and systematically, and the transmission speed can be improved. By controlling the transmission nodes to remain in the optimal transmission group state, the data transmission efficiency can be further improved and resource utilization can be accelerated.

[0020] Step 4: Analyze data relationships to build a knowledge graph, use the positioning module to locate the data in the data tree database, identify the data carrier mark, and identify the data source.

[0021] In an embodiment of the present invention, the types of data clusters are identified, the relationships between the data in each cluster are identified, and a knowledge graph is constructed through a knowledge graph construction module; Furthermore, at each procedure that the data goes through, the data marking module marks the processing method of the data on the data carrier, and compares the original data after each procedure. If the comparison results are completely consistent, the next procedure will be entered. If there are differences in the comparison results, when the similarity is greater than a threshold, the system obtains the original data and continues the procedure. When the similarity is less than a threshold, the system screens out the data and hands it over to manual processing; Furthermore, the data before the difference in the comparison result is located, the data before the difference is obtained and re-entered into the program for processing, and the process and source of the data are traced according to the mark on the data carrier. By analyzing the data relationship and constructing a knowledge graph, it is more conducive to discovering the utilization value and hidden value of the data and improving the utilization rate of the data. By marking the data in each program, the processing links and the source of the data can be traced back to achieve data traceability.

[0022] Embodiment 2: Embodiment 2 of the present invention provides an Internet-based data management and control system. Figure 2 A schematic diagram of the module composition of an Internet-based data management and control system provided in Embodiment 2 of the present invention is shown in FIG. Figure 2 As shown, the system includes: Data collection module, used to periodically pull data from the enterprise business system; The data processing module is used to process the data and prioritize the data through the data rating model; The data control module is used to control and manage data transmission nodes and trace abnormal data.

[0023] In some embodiments of the present invention, the data acquisition module includes: A data collection module is used to periodically pull data through a data interface; A data transmission module, used to convert electrical signals into digital signals for data transmission; The database module is used to establish a database for the data pulled in each cycle.

[0024] In some embodiments of the present invention, the data processing module includes: Standardization processing module, used to eliminate incomplete, erroneous, and redundant data from the collected data, and convert the data into data with a unified format; Data classification module, used to analyze data types and cluster data; Model training module, used to retrieve historical data from the historical database to train the data model; The data rating module is used to assess the transmission level of data.

[0025] In some embodiments of the present invention, the data control module includes: A node processing module is used to manage transmission nodes; Knowledge graph construction module, used to build knowledge graphs between data; Data labeling module, used to label the data source; Data location module, used to locate the data in the knowledge graph; The data tracing module is used to trace the source of data and the process that the data has gone through.

[0026] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0027] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data management and control system based on the Internet, characterized in that: The method comprises the following steps: Step 1: Establish a database, back up and store data, collect enterprise data, mark data carriers, and transfer enterprise data to the management and control system; Step 2: Standardize the collected data, classify the processed data, and extract the training data set for model training; Step 3: Use the trained model to rate the data, and the transmission node selects a transmission node to transmit the data based on the rating results; Step 4: Analyze data relationships to build a knowledge graph, use the positioning module to locate the data in the data tree database, identify the data carrier mark, and identify the data source.

2. The Internet-based data management and control system according to claim 1, characterized in that: The steps of establishing a database, backing up and storing data, collecting enterprise data, marking data carriers, and transmitting enterprise data to the management and control system include: The system uses a data interface to connect to the enterprise business system and periodically pulls data from the enterprise business system through a carrier. When the carrier pulls data, the data marking module recognizes that the system is connected to the database at this time and marks the database information on the carrier. The database module establishes a database for the data obtained for each cycle, and transmits the data to the database through the data transmission module. The data is backed up and stored in the database. The data transmission module converts the electrical signal for data transmission into a digital signal, and uses the digital signal for data transmission.

3. The Internet-based data management and control system according to claim 2, characterized in that: The step of standardizing the collected data and extracting a training data set for model training includes: The standardization processing module performs standardization processing on the data, calls the first data standardization model to eliminate incomplete data, erroneous data, and duplicate data, and establishes a review database for the eliminated data to store it, connects to the review database through the data interface to obtain data, manually reviews the eliminated data, supplements the incomplete data, corrects the erroneous data, and standardizes the manually reviewed data again; Further, the second standardized model is called to process the data after cluster classification, identify the data structure of each cluster, anchor the unstructured data and semi-structured data, identify the data type in the cluster, and determine whether the data can be converted into structured data. When the data type cannot be converted into structured data, the second standardized model retains the data type. When the data type can be converted into structured data, the second standardized model converts the data into structured data. The model training module extracts historical data from the database as a training data set to train the data rating model, trains the data rating model, and adds a special rating recognition function after the data rating model training is completed to recognize the expedited transmission mark added to the data carrier.

4. The Internet-based data management and control system according to claim 3, characterized in that: The step of classifying the processed data comprises: Acquire the data after the first standardization processing, perform clustering processing on the data, cluster the data into structured data, semi-structured data and unstructured data, analyze the data in each cluster based on the clustering, identify the data type, classify the data according to the identified data type, and mark the data type in each classified data set.

5. The Internet-based data management and control system according to claim 4, characterized in that: The step of rating the data using the trained model and the transmission node selecting a transmission node to transmit the data according to the rating result includes: The data rating module obtains the data to be transmitted, calls the rating model to assess the transmission priority of the data to be transmitted, and the rating model identifies the specific data priority transmission mark on the data carrier. The rating model assesses the data as the first transmission echelon level. If the data priority transmission mark is not identified, the rating training model is used to assess the data transmission level according to the training rules. Furthermore, the rating model transmits a message that the data to be transmitted is in the first transmission echelon level to the node management module. The node management module receives the message and starts the dedicated transmission node for expedited data to preheat the node. The expedited transmission data is transmitted through its dedicated transmission node. Furthermore, when the transmission node is receiving data for data transmission, the node management module monitors in real time the amount of remaining data to be transmitted on each transmission node. When the ratio of the remaining amount of data to be transmitted to the transmission time of the transmission node is less than a preset threshold of the system, the transmission node enters a transmission fatigue state, and the transmission speed decreases. The node management module starts a new transmission node to transmit the remaining data through the new transmission node, and the transmission node that enters the fatigue state will be shut down.

6. The Internet-based data management and control system according to claim 5, characterized in that: The steps of analyzing data relationships to construct a knowledge graph, using a positioning module to locate the position of data in a data tree database, identifying a data carrier tag, and identifying a data source include: Identify the types of data clusters, identify the relationships between data in each cluster, and build a knowledge graph through the knowledge graph construction module; Furthermore, at each procedure that the data goes through, the data marking module marks the processing method of the data on the data carrier, and compares the original data after each procedure. If the comparison results are completely consistent, the next procedure will be entered. If there are differences in the comparison results, when the similarity is greater than a threshold, the system obtains the original data and continues the procedure. When the similarity is less than a threshold, the system screens out the data and hands it over to manual processing; Furthermore, the data before the difference in the comparison result is located, the data before the difference is obtained and re-entered into the program for processing, and the process the data has gone through and the source of the data are traced back according to the mark on the data carrier.

7. An Internet-based data management and control system, characterized in that: The system comprises: Data collection module, used to periodically pull data from the enterprise business system; The data processing module is used to process the data and prioritize the data through the data rating model; The data control module is used to control and manage data transmission nodes and trace abnormal data.

8. The Internet-based data management and control system according to claim 7, characterized in that: The data acquisition module comprises: A data collection module is used to periodically pull data through a data interface; A data transmission module, used to convert electrical signals into digital signals for data transmission; The database module is used to establish a database for the data pulled in each cycle.

9. The Internet-based data management and control system according to claim 8, characterized in that: The data processing module comprises: Standardization processing module, used to eliminate incomplete, erroneous, and redundant data from the collected data, and convert the data into data with a unified format; Data classification module, used to analyze data types and cluster data; Model training module, used to retrieve historical data from the historical database to train the data model; The data rating module is used to assess the transmission level of data.

10. The Internet-based data management and control system according to claim 9, characterized in that: The data control module comprises: A node processing module is used to manage transmission nodes; Knowledge graph construction module, used to build knowledge graphs between data; Data labeling module, used to label the data source; Data location module, used to locate the data in the knowledge graph; The data tracing module is used to trace the source of data and the process that the data has gone through.

Images