An internet-based data management and control system and a management and control method thereof

Abstract

The application discloses a kind of based on internet's data management and control system's management and control method, the method includes the following steps: step one: establish database, backup and store data, collect enterprise data, and mark data carrier, transmit enterprise data to management and control system;Step two: the data collected is standardized, and the data processed is classified, and the training data set is extracted to train model;Step three: the model is trained to the data rating, and the transmission node selects transmission node according to rating result and transmits data;Step four: analysis data relationship constructs knowledge graph, utilizes positioning module and positions data in data tree database, identifies data carrier mark, identifies data source.The application has the characteristics of improving data utilization efficiency and traceable data source.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically to a management method for an Internet-based data management system. Background Technology

[0002] With the rapid development of internet technology, internet data is experiencing explosive growth. Enterprises need to process and manage vast amounts of data for effective utilization. However, current technologies still need to improve data utilization efficiency. While existing technologies can train models to rate data and determine transmission order, not all test data collected during model training meets the model's rating criteria. Some data cannot be rated by the training model, and the data ultimately stored in the database lacks traceability to ensure the accuracy of its source. Therefore, designing an internet-based data management system that improves data utilization efficiency and ensures traceability of data sources is essential. Summary of the Invention

[0003] The purpose of this invention is to provide a management and control method for an Internet-based data management and control system, so as to solve the problems mentioned in the background art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a management and control method for an internet-based data management and control system, comprising the following steps:

[0005] Step 1: Establish a database, back up and store the data, collect enterprise data, label the data carrier, and transmit the enterprise data to the management and control system;

[0006] Step 2: Standardize the collected data, classify the processed data, and extract the training dataset for model training;

[0007] Step 3: Use the trained model to rate the data, and the transmission nodes select transmission nodes to transmit the data based on the rating results;

[0008] Step 4: Analyze data relationships to construct a knowledge graph, use the positioning module to locate the data in the data tree database, identify the data carrier tags, and identify the data source.

[0009] According to the above technical solution, the steps of establishing a database, backing up and storing data, collecting enterprise data, marking data carriers, and transmitting enterprise data to the management and control system include:

[0010] The system connects to the enterprise business system through a data interface and periodically pulls data from the enterprise business system through a carrier. When the carrier pulls data, the data tagging module identifies that the system is connected to the database and tags the database information on the carrier. The database module builds a database for the data acquired in each period and transmits the data to the database through the data transmission module. The data is backed up and stored in the database. The data transmission module converts the electrical signals used for data transmission into digital signals and uses the digital signals for data transmission.

[0011] According to the above technical solution, the steps of standardizing the collected data and extracting the training dataset for model training include:

[0012] The standardization processing module standardizes the data, retrieves the first data standardization model to remove incomplete, erroneous, and duplicate data, and establishes a review database for storing the removed data. The module connects to the review database through a data interface to obtain data, and the removed data is manually reviewed to supplement incomplete data and correct erroneous data. The manually reviewed data is then standardized again.

[0013] Furthermore, the second standardization model is invoked to process the clustered data, identify the data structure of each cluster, anchor unstructured and semi-structured data, identify the data types in the cluster, and determine whether the data can be converted into structured data. When a data type cannot be converted into structured data, the second standardization model retains the data type. When a data type can be converted into structured data, the second standardization model converts the data into structured data.

[0014] The model training module extracts historical data from the database as a training dataset to train the data rating model. After the data rating model is trained, a special rating and recognition function is added to identify the expedited transmission mark added to the data carrier.

[0015] According to the above technical solution, the step of classifying the processed data includes:

[0016] After obtaining the data that has undergone the first standardization process, the data is clustered into structured data, semi-structured data, and unstructured data. Based on the clustering, the data in each cluster is analyzed to identify the data types. The data is then classified according to the identified data types, and the data types are labeled in each classified data set.

[0017] According to the above technical solution, the step of using a trained model to rate the data, and the transmission node selecting a transmission node to transmit the data based on the rating result, includes:

[0018] The data rating module acquires the data to be transmitted, calls the rating model to assess the transmission priority of the data to be transmitted, the rating model identifies specific data priority transmission identifiers on the data carrier, and the rating model rates the data as the first transmission tier level. If no data priority transmission identifier is identified, the rating training model is used to assess the data transmission level according to the training rules.

[0019] Furthermore, the rating model transmits the message that the data to be transmitted belongs to the first transmission tier to the node management module. The node management module receives the message and starts the dedicated transmission node for urgent data to warm up the node. The urgent data is transmitted through its dedicated transmission node.

[0020] Furthermore, during the data transmission process, the node management module monitors the amount of data remaining to be transmitted at each transmission node in real time. When the ratio of the remaining amount of data to be transmitted to the transmission time of the transmission node is less than the system's preset threshold, the transmission node enters a transmission fatigue state, and the transmission speed decreases. The node management module then starts a new transmission node to transmit the remaining data through the new transmission node, and the transmission node that has entered the fatigue state will be shut down.

[0021] According to the above technical solution, the steps of analyzing data relationships to construct a knowledge graph, using a positioning module to locate the data's position in the data tree database, identifying data carrier tags, and identifying the data source include:

[0022] Identify data cluster types, identify the relationships between data in each cluster, and construct a knowledge graph using the knowledge graph construction module;

[0023] Furthermore, during each process the data goes through, the data tagging module marks the processing method of the data on the data carrier, and compares the original data after each process. If the comparison results match perfectly, the next process is initiated. If there are differences in the comparison results, when the similarity is greater than the threshold, the system retrieves the original data and continues the process. When the similarity is less than the threshold, the system filters out the data and hands it over to manual processing.

[0024] Furthermore, the location data is the data before the difference appears in the comparison results. The data before the difference appears is retrieved and re-entered into the program for processing. The process and source of the data are traced based on the markings on the data carrier.

[0025] According to the above technical solution, the system includes:

[0026] The data acquisition module is used to periodically retrieve data from the enterprise's business systems;

[0027] The data processing module is used to process data and prioritize it using a data rating model.

[0028] The data control module is used to control and manage data transmission nodes and to trace abnormal data.

[0029] According to the above technical solution, the data acquisition module includes:

[0030] The data collection module is used to periodically retrieve data through a data interface;

[0031] The data transmission module is used to convert electrical signals into digital signals for data transmission.

[0032] The database module is used to create a database for the data retrieved in each period.

[0033] According to the above technical solution, the data processing module includes:

[0034] The standardization processing module is used to remove incomplete, erroneous, and redundant data from the collected data and transform the data into data with a uniform format.

[0035] The data classification module is used to analyze data types and perform clustering processing on the data.

[0036] The model training module is used to retrieve historical data from the historical database to train the data model;

[0037] The data rating module is used to assess the transmission quality of data.

[0038] According to the above technical solution, the data control module includes:

[0039] The node processing module is used to manage transmission nodes;

[0040] The knowledge graph construction module is used to build knowledge graphs between data.

[0041] The data tagging module is used to tag the data source;

[0042] The data location module is used to locate the position of data in the knowledge graph;

[0043] The data traceability module is used to trace and find the source of data and the process that the data has gone through.

[0044] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention, by setting up a standardized processing module and a data model training module, can eliminate incomplete, erroneous, and duplicate data through standardized data processing, and ensure data integrity through manual review. By converting unstructured and semi-structured data into structured data, it can guarantee data uniformity. On this basis, it can improve the speed of model training, improve the accuracy of the trained model's judgment, and thus improve data utilization efficiency. By classifying data, it can transmit data quickly and systematically, improving transmission speed. By controlling the transmission nodes to maintain the optimal transmission group state, it can further improve data transmission efficiency and accelerate resource utilization. By analyzing data relationships and constructing a knowledge graph, it is easier to discover the utilization value and hidden value of data, improving data utilization rate. By marking data in various processes, it is possible to trace the processing stages and source of data, achieving data traceability. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart of the steps of a data management and control method based on the Internet provided in Embodiment 1 of the present invention;

[0047] Figure 2 This is a schematic diagram of the module composition of an Internet-based data management and control system provided in Embodiment 2 of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1: Figure 1 This is a flowchart of an Internet-based data management and control system provided in Embodiment 1 of the present invention. This embodiment can be applied to data management scenarios, and the method can be executed by the Internet-based data management and control system provided in this embodiment, such as... Figure 1 As shown, the method specifically includes the following steps:

[0050] Step 1: Establish a database, back up and store the data, collect enterprise data, label the data carrier, and transmit the enterprise data to the management and control system;

[0051] In this embodiment of the invention, the system connects to the enterprise business system via a data interface and periodically pulls data from the enterprise business system through a carrier. When the carrier pulls data, the data marking module identifies that the system is connected to the database and marks the database information on the carrier. The database module establishes a database for the data acquired in each period and transmits the data to the database through the data transmission module. The data is backed up and stored in the database. The data transmission module converts the electrical signals used for data transmission into digital signals and uses the digital signals for data transmission. By establishing a database for the data acquired in each period, it is possible to ensure that the database can use the types of data collected, without causing data loss or data incompleteness, thus protecting the integrity of the data. Marking the data carrier can ensure the data source, enabling the determination of the data source in the event of a system failure, tracing back to complete historical data, and ensuring the authenticity and integrity of the data. Converting electrical signals into digital signals can better resist interference and reduce interference encountered by the data during transmission.

[0052] Step 2: Standardize the collected data, classify the processed data, and extract the training dataset for model training;

[0053] In this embodiment of the invention, the standardization processing module performs standardization processing on the data, retrieves the first data standardization model to remove incomplete data, erroneous data, and duplicate data, and establishes a review database for storing the removed data. The review database is connected through a data interface to obtain data, and the removed data is manually reviewed to supplement incomplete data and correct erroneous data. The manually reviewed data is then standardized again.

[0054] Furthermore, the data after the first standardization process is obtained, and the data is clustered into structured data, semi-structured data and unstructured data. Based on the clustering, the data in each cluster is analyzed to identify the data type. The data is classified according to the identified data type, and the data type is labeled in each classified data set.

[0055] Furthermore, the second standardization model is invoked to process the clustered data, identify the data structure of each cluster, anchor unstructured and semi-structured data, identify the data types in the cluster, and determine whether the data can be converted into structured data. When a data type cannot be converted into structured data, the second standardization model retains the data type. When a data type can be converted into structured data, the second standardization model converts the data into structured data.

[0056] Furthermore, the model training module extracts historical data from the database as a training dataset to train the data rating model. After the data rating model is trained, a special rating and recognition function is added to identify the expedited transmission markers added to the data carrier. By standardizing the data, incomplete, erroneous, and duplicate data can be removed. Furthermore, the data is manually reviewed to ensure its integrity. By converting unstructured and semi-structured data into structured data, data consistency can be guaranteed. On this basis, the training speed of the model can be improved, the accuracy of the trained model can be improved, and thus the efficiency of data utilization can be improved.

[0057] Step 3: Use the trained model to rate the data, and the transmission nodes select transmission nodes to transmit the data based on the rating results;

[0058] In this embodiment of the invention, the data rating module acquires the data to be transmitted, calls the rating model to evaluate the transmission priority of the data to be transmitted, the rating model identifies a specific data priority transmission identifier on the data carrier, and the rating model rates the data as the first transmission tier level. If no data priority transmission identifier is identified, the rating training model is used to evaluate the data transmission level according to the training rules.

[0059] Furthermore, the rating model transmits the message that the data to be transmitted belongs to the first transmission tier to the node management module. The node management module receives the message and starts the dedicated transmission node for urgent data to warm up the node. The urgent data is transmitted through its dedicated transmission node.

[0060] Furthermore, during the data transmission process, the node management module monitors the amount of remaining data to be transmitted at each transmission node in real time. When the ratio of the remaining amount of data to be transmitted to the transmission time of the transmission node is less than the system's preset threshold, the transmission node enters a transmission fatigue state, and the transmission speed decreases. The node management module then starts a new transmission node to transmit the remaining data through the new transmission node, and the transmission node in the fatigue state will be shut down. By classifying the data, the data can be transmitted quickly and systematically, improving the transmission speed. By controlling the transmission nodes to remain in the optimal transmission group state, the data transmission efficiency is further improved, and resource utilization is accelerated.

[0061] Step 4: Analyze data relationships to construct a knowledge graph, use the positioning module to locate the data in the data tree database, identify the data carrier tags, and identify the data source.

[0062] In this embodiment of the invention, the types of data clusters are identified, the relationships between data in each cluster are identified, and a knowledge graph is constructed using a knowledge graph construction module.

[0063] Furthermore, during each process the data goes through, the data tagging module marks the processing method of the data on the data carrier, and compares the original data after each process. If the comparison results match perfectly, the next process is initiated. If there are differences in the comparison results, when the similarity is greater than the threshold, the system retrieves the original data and continues the process. When the similarity is less than the threshold, the system filters out the data and hands it over to manual processing.

[0064] Furthermore, the system locates the data before the discrepancy appears in the comparison results, retrieves the data before the discrepancy, and re-enters the program for processing. Based on the markings on the data carrier, the process and source of the data are traced. By analyzing the data relationships and constructing a knowledge graph, it is easier to discover the utilization value and hidden value of the data, thereby improving the data utilization rate. By marking the data in each program, the processing links and source of the data can be traced, thus achieving data traceability.

[0065] Example 2: Example 2 of the present invention provides an Internet-based data management and control system. Figure 2 This is a schematic diagram of the module composition of an Internet-based data management and control system provided in Embodiment 2 of the present invention, as shown below. Figure 2 As shown, the system includes:

[0066] The data acquisition module is used to periodically retrieve data from the enterprise's business systems;

[0067] The data processing module is used to process data and prioritize it using a data rating model.

[0068] The data control module is used to control and manage data transmission nodes and to trace abnormal data.

[0069] In some embodiments of the present invention, the data acquisition module includes:

[0070] The data collection module is used to periodically retrieve data through a data interface;

[0071] The data transmission module is used to convert electrical signals into digital signals for data transmission.

[0072] The database module is used to create a database for the data retrieved in each period.

[0073] In some embodiments of the present invention, the data processing module includes:

[0074] The standardization processing module is used to remove incomplete, erroneous, and redundant data from the collected data and transform the data into data with a uniform format.

[0075] The data classification module is used to analyze data types and perform clustering processing on the data.

[0076] The model training module is used to retrieve historical data from the historical database to train the data model;

[0077] The data rating module is used to assess the transmission quality of data.

[0078] In some embodiments of the present invention, the data control module includes:

[0079] The node processing module is used to manage transmission nodes;

[0080] The knowledge graph construction module is used to build knowledge graphs between data.

[0081] The data tagging module is used to tag the data source;

[0082] The data location module is used to locate the position of data in the knowledge graph;

[0083] The data traceability module is used to trace and find the source of data and the process that the data has gone through.

[0084] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0085] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A management and control method for an Internet-based data management and control system, characterized in that: Includes the following steps: Step 1: Establish a database, back up and store the data, collect enterprise data, label the data carriers, and transmit the enterprise data to the management and control system; including: The system connects to the enterprise business system through a data interface and periodically pulls data from the enterprise business system through a carrier. When the carrier pulls data, the data marking module identifies that the system is connected to the database and marks the database information on the carrier. The database module builds a database for the data acquired in each period and transmits the data to the database through the data transmission module. The data is backed up and stored in the database. The data transmission module converts the electrical signals used for data transmission into digital signals and uses the digital signals for data transmission. Step Two: Standardize the collected data, classify the processed data, and extract the training dataset for model training; including: The standardization processing module standardizes the data, retrieves the first data standardization model to remove incomplete, erroneous, and duplicate data, and establishes a review database for storing the removed data. The module connects to the review database through a data interface to obtain data, and the removed data is manually reviewed to supplement incomplete data and correct erroneous data. The manually reviewed data is then standardized again. After obtaining the data after the first standardization process, the data is clustered into structured data, semi-structured data and unstructured data. Based on the clustering, the data in each cluster is analyzed to identify the data type. The data is then classified according to the identified data type, and the data type is labeled in each classified data set. The second standardization model is invoked to process the clustered data, identify the data structure of each cluster, anchor unstructured and semi-structured data, identify the data types in the cluster, and determine whether the data can be converted into structured data. When the data type cannot be converted into structured data, the second standardization model retains the data type. When the data type can be converted into structured data, the second standardization model converts the data into structured data. The model training module extracts historical data from the database as a training dataset to train the data rating model. After the data rating model is trained, a special rating and recognition function is added to identify the expedited transmission mark added to the data carrier. Step 3: The trained model is used to rate the data, and the transmission nodes select transmission nodes to transmit the data based on the rating results; this includes: The data rating module acquires the data to be transmitted, calls the rating model to assess the transmission priority of the data to be transmitted, the rating model identifies specific data priority transmission identifiers on the data carrier, and the rating model rates the data as the first transmission tier level. If no data priority transmission identifier is identified, the rating training model is used to assess the data transmission level according to the training rules. The rating model transmits the message that the data to be transmitted belongs to the first transmission tier to the node management module. The node management module receives the message and starts the dedicated transmission node for urgent data to warm up the node. The urgent data is transmitted through its dedicated transmission node. During the data transmission process, the node management module monitors the amount of data remaining to be transmitted at each transmission node in real time. When the ratio of the remaining amount of data to be transmitted to the transmission time of the transmission node is less than the system's preset threshold, the transmission node enters a transmission fatigue state, and the transmission speed decreases. The node management module then starts a new transmission node to transmit the remaining data through the new transmission node, and the transmission node that has entered the fatigue state will be shut down. Step 4: Analyze data relationships to construct a knowledge graph, use the positioning module to locate the data's position in the data tree database, identify data carrier tags, and identify the data source; including: Identify the types of data clusters, identify the relationships between data in each cluster, and construct a knowledge graph using the knowledge graph construction module; During each process the data goes through, the data tagging module marks the processing method on the data carrier and compares it with the original data after each process. If the comparison results match perfectly, the next process is initiated. If there are differences in the comparison results, when the similarity is greater than the threshold, the system retrieves the original data and continues the process. When the similarity is less than the threshold, the system filters out the data and hands it over to manual processing. The location data is the data before the difference appears in the comparison results. The data before the difference appears is retrieved and re-entered into the program for processing. The process and source of the data are traced based on the markings on the data carrier.

Images